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Foreword 


!  am  proud  to  present  the  first  edition  of  The  NATO  Lessons  Learned  Handbook,  an 
important  resource  for  everyone  involved  in  helping  NATO  and  its  partners  to  learn 
effectively  from  experience. 

The  NATO  Lessons  Learned  Handbook  is  an  easy  to  use,  practical  guide  to  NATO’s 
approach  to  Lessons  Learned.  Although  it  is  based  on  NATO’s  approach,  I  believe  it  is 
also  a  valuable  resource  for  anyone  implementing  Lessons  Learned  in  any 
organization. 

Each  edition  of  The  NATO  Lessons  Learned  Handbook  will  incorporate  the  latest 
experience  and  expertise  of  the  NATO  Lessons  Learned  community.  This  edition  is  no 
exception.  I  hope  you  will  keep  this  handbook  with  you,  so  that  you  can  benefit  from 
the  experience  of  others  in  your  every  day  work  and,  in  doing  so,  demonstrate  a 
commitment  to  Lessons  Learned  in  your  personal  activity  as  well  as  your  organizational 
responsibilities. 
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Introduction 

Welcome  to  the  NATO  Lessons  Learned  (LL)  Handbook.  The  purpose  of  this  handbook 
is  to  guide  and  assist  you  to  fulfill  your  role  in  supporting  your  organization  and  NATO 
to  effectively  learn  from  experience.  This  handbook  uses  NATO’s  approach  to  LL  as 
the  underlying  model  but  has  been  written  to  be  relevant  to  any  organization. 

This  opening  chapter  sets  the  stage  for  the  rest  of  this  handbook  by  giving  an  overview 
of: 

•  What  Is  LL? 

•  What  Is  The  Role  Of  A  LL  Staff  Officer  (LLSO)? 

•  How  To  Get  Started  In  LL? 

WHAT  IS  LESSONS  LEARNED? 

LL  is  broadly  used  to  describe  people,  things  and  activities  related  to  the  act  of  learning 
from  experience  to  achieve  improvements.  The  idea  of  LL  in  an  organization  is  that 
through  a  formal  approach  to  learning,  individuals  and  the  organization  can  reduce  the 
risk  of  repeating  mistakes  and  improve  the  chance  that  successes  are  repeated.  In  the 
military  context,  this  means  reduced 
operational  risk,  lower  cost  and 
improved  operational  effectiveness. 

LL  as  a  deliberate  approach  to 
improve  an  organization’s  learning  is 
sometimes  criticized  as  being  an 
administrative  burden  without 
recognizing  the  value  it  delivers. 

However,  if  LL  is  set  up  and  run 
properly,  it  can  provide  enormous 
value. 

Lessons  can  be  derived  from  any  activity.  They  are  a  product  of  operations,  exercises, 
training  and  experiments.  During  the  course  of  our  activities  most  of  us  will  recognize 
ways  of  doing  things  more  easily  or  efficiently  that  can  be  passed  on  to  our  colleagues 
and  successors  to  help  them  avoid  problems  and  do  even  better  than  we  did. 

LL  describes  more  than  just  learning  from  experience.  Learning  must  be  used  to  justify 
changes  that  will  lead  to  improved  performance.  This  is  made  clear  in  NATO’s  Allied 
Joint  Doctrine  for  Joint  Operations,  which  states: 

“The  purpose  of  a  Lessons  Learned  procedure  is  to  learn  efficiently  from 
experience  and  to  provide  validated  justifications  for  amending  the  existing  way  of 
doing  things,  in  order  to  improve  performance,  both  during  the  course  of  an 
operation  and  for  subsequent  operations.  This  requires  lessons  to  be  meaningful 
and  for  them  to  be  brought  to  the  attention  of  the  appropriate  authority  able  and 
responsible  for  dealing  with  them.  It  also  requires  the  chain  of  command  to  have  a 
clear  understanding  of  how  to  prioritise  lessons  and  how  to  staff  them.”  AJP-3(A) 

Allied  Doctrine  for  Joint  Operations1 


Definition:  Lessons  Learned  or  Lesson_  Learned 

Lessons  Learned,  an  adjective,  describes  anything 
related  to  a  Lessons  Learned  procedure.  E.g. 
Lessons  Learned  process,  Lessons  Learned  staff 
officer,  Lessons  Learned  working  group,  etc. 

•  Lesson_  Learned,  a  noun,  a  written  record  of  the 
final  lesson,  the  result  of  the  Lessons  Learned 
procedure. 

In  this  handbook,  LL  is  only  used  as  an  abbreviation  for 
Lessons  Learned  when  used  as  an  adjective.  Other 
publications  may  use  LL  to  abbreviate  Lesson  Learned 
the  noun.  Careful! 


1  Allied  Doctrine  for  Joint  Operations,  AJP-3(A),  July  2007,  NATO/PfP  Unclassified. 
Paragraph  0451 . 
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Depending  on  your  nation,  HQ  or  organization,  you  will  encounter  the  term  Lesson 
Learned  applied  both  to  the  end  product  of  an  LL  process  or  to  one  of  the  intermediate 
products,  i.e.  a  lesson  to  be  learned. 

•  A  NATO  Lesson  Learned  “Results  from  the  implementation  of  a  remedial  action 
that  produced  an  improved  performance  or  increased  capability.  ”  NAT O  LL 
Policy  (Reference  A) 

To  create  a  Lesson  Learned  you  will  capture  and  exploit  both  explicit  and  tacit 
knowledge: 

•  Explicit  Knowledge  is  knowledge  that  has  been  documented.  This  type  of 
knowledge  can  lead  to  a  Lesson  Learned  by  the  use  of  an  LL  process  and  LL 
information  sharing  tools  such  as  databases  and  wikis. 

•  Tacit  Knowledge  is  knowledge  that  has  not  been  documented  but  is  still 
extremely  valuable.  This  type  of  knowledge  is  stored  in  our  heads  and  can  lead 
to  a  Lesson  Learned  when  we  interact  with  others  by  discussion  and  sharing 
experience  within  a  community,  perhaps  facilitated  by  formal  working  groups, 
training  courses,  conferences  or  other  events. 

Regardless  of  whether  you  are  learning  from  explicit  or  tacit  knowledge,  in  any  learning 
organization,  you  will  follow  the  same  three  basic  stages  of  learning.  These  are  as 
described  in  The  Lessons  Learned  Handbook  (Reference  B)2: 

1.  Identification:  Collect  learning  from  experiences. 

2.  Action:  Take  action  to  change  existing  ways  of  doing  things  based  on  the 
learning. 

3.  Institutionalization:  Communicate  the  change  so  that  relevant  parts  of  the 
organization  can  benefit  from  the  learning. 

For  example,  consider  how  the  Counter  Improvised  Explosive  Device  (IED)  community 
learns: 

1.  Identification:  After  every  IED  incident  a  report  is  generated  that  identifies  what 
can  be  learned  from  the  incident. 

2.  Action:  The  reports  are  reviewed  by  national  and  multinational  groups  who  take 
the  necessary  action  to  learn  from  the  experience.  Usually  this  is  an  update  to 
doctrine,  Standing/Standard  Operating  Procedures  (SOP)  or  tactics,  techniques 
and  procedures. 

3.  Institutionalization:  The  new  procedures  are  incorporated  into  training  for  new 
staff  and  communicated  to  current  staff  through  newsletters  and  bulletins. 

The  activities  NATO  uses  to  promote  learning  from  experience  vary  across 
organizations  and  usually  include: 

•  LL  process:  To  gather,  staff,  action  and  communicate  lessons  to  ensure  learning 
from  experience  is  converted  into  actual  improvement  via  a  formal  process. 

•  LL  information  sharing:  To  make  use  of  databases,  spreadsheets,  websites, 
reports  or  other  media  to  store  and  communicate  lessons. 


2  These  three  stages  are  generic  and  apply  to  any  learning  organization:  NATO,  military,  civilian 
(government  or  private  enterprise).  They  are  not  yet  explicitly  embedded  in  NATO  doctrine. 
‘Institutionalization’,  in  NATO  would  be  Lesson  sharing  and,  as  applicable,  incorporation  into 
NATO  doctrine  and  procedures. 
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•  LL  community:  To  bring  together  Subject  Matter  Experts  (SME)  at  working 
groups,  training  courses,  conferences  and  other  events  to  share  experience  and 
learning. 

WHAT  IS  THE  ROLE  OF  A  LESSONS  LEARNED  STAFF  OFFICER? 

Everyone  in  an  organization  has  a  responsibility  for  LL,  but  the  LLSO  is  central  to  the 
organization’s  efforts  to  engage  everybody  in  seeing  the  value  of  lessons  learned.  If 
people  are  not  engaged,  they  see  no  value  and  do  not  actively  participate.  It  then 
becomes  impossible  for  learning  to  take  place  and  a  ‘vicious  circle’  is  created  nullifying 
the  LL  process. 

The  simplest  way  to  get  everyone  involved  in  LL  is  to  ensure  that  LL  in  your 
organization  is  constantly  demonstrating  value.  The  LLSO  has  an  important  role  to  play 
in  conducting  staff  work  to  support  the  organization’s  LL  process,  LL  information 
sharing  and  participation  in  the  LL  community.  The  LLSO  may  also  need  to  set  up  or 
improve  the  organization’s  LL  capability. 

All  organizations  will  have  LL  procedures  and  tools  that  are  tailored  to  their  needs.  The 
LLSO  role  will  be  defined  within  this  context.  Typically  an  LLSO  will  be  expected  to  do 
the  following: 

Support  the  Lessons  Learned  Process  (See  Chapters  2-5) 

•  Gather  the  organization’s  observations  on  a  regular  basis  and  immediately  after 
every  activity  the  organization  undertakes  such  as  missions,  training  events  or 
exercises. 

•  Analyse  observations  collected  to  establish  whether  action  needs  to  be  taken  to 
learn  and  if  so,  what  action  is  needed. 

•  Present  Lessons  Identified  (LI)  to  the  command  group  so  they  can  make  a 
decision  about  what  action  to  take. 

•  Keep  track  of  the  progress  of  action  relating  to  Lis  and  keep  the  command  group 
up-to-date  on  which  LI  have  become  lessons  learned. 

•  Organize  meetings  for  the  staffing  of  Lis. 

Support  Lessons  Learned  Information  Sharing  (See  Chapter  6) 

•  Maintain  a  store  of  the  organization’s  LL  information. 

•  Prepare  and  release  the  organization’s  LL  information  to  other  organizations  as 
appropriate. 

•  Enter  the  organization’s  observations,  Lis  and  lessons  learned  into  the  NATO  LL 
Database  (LLDb)  as  appropriate. 

•  Provide  LL  information  in  response  to  requests  from  inside  the  organization  (e.g. 
from  the  planning  cell  or  trainers)  or  from  outside  the  organization. 

•  Compile  regular  newsletters/summaries  of  LL  information  to  keep  people  up-to- 
date. 

Support  the  Lessons  Learned  Community  (See  Chapter  6) 

•  Attend  the  NATO  (or  other  relevant)  LL  Conference. 

•  Organize  working  groups/training  events  for  knowledge  sharing. 

•  Represent  the  organization  in  relevant  LL  sharing  events. 

•  Maintain  a  Point  of  Contact  (POC)  list  of  SMEs  in  the  organization. 
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•  Keep  LLSO  contact  details  up-to-date  in  the  NATO  LL  POC  Database. 

Support  the  LL  Capability 

The  LL  capability  within  the  organization  may  still  need  to  be  established  or  improved 
upon  and  this  job  will  fall  to  the  LLSO.  Everything  in  this  handbook  will  help  an  LLSO 
to  set  up  an  LL  capability  that  is  relevant  and  effective  for  the  organization  and  is 
compliant  with  NATO’s  overall  LL  approach.  A  checklist  of  suggested  LL  capability 
features  (Annex  E)  may  help  you  assess  your  current  LL  capability. 

HOW  TO  GET  STARTED  IN  LESSONS  LEARNED? 

You  are  already  on  the  right  track  having  read  this  far  through  The  NATO  LL 
Handbook.  However,  this  handbook  is  not  the  only  resource  available  to  help  you  get 
started.  The  following  NATO  resources  are  available: 

Fundamental  NATO  Lessons  Learned  References 

You  should  read  and  familiarize  yourself  with  the  fundamental  NATO  LL  references: 

•  NATO  Lessons  Learned  Policy  (Reference  A). 

•  Bi-Strategic  Command  (Bi-SC)  Directive  80-6  Lessons  Learned  (Reference  C). 

•  Allied  Command  Operations  (ACO)  Directive  80-1  Lessons  Learned  (Reference 
D). 

NATO  LL  Policy  is  applicable  to  all  NATO  bodies,  agencies  and  staffs,  and  acts  as  a 
guide  to  Allies  and  other  international  organizations.  It  establishes  the  basic  principles 
of  an  Alliance-wide  approach  to  LL  in  order  to  ensure  transparency  and  a  common 
understanding  of  its  intent. 

Bi-SC  Directive  80-6  Lessons  Learned  is  applicable  to  all  levels  of  the  NATO 
Command  Structure,  and  HQs  in  the  NATO  Force  Structure  participating  in  NATO 
operations  and  exercises.  It  establishes  a  standardized  and  coordinated  approach  to 
the  NATO  LL  process  and  details  tasks  and  responsibilities  at  the  strategic  level. 

ACO  Directive  80-1  Lessons  Learned  is  applicable  to  all  levels  of  ACO  HQs  and  to 
SHAPE  divisions.  It  provides  additional  guidance  and  direction  for  the  implementation 
of  the  LL  process  within  ACO  and  is  a  supplement  to  the  Bi-SC  Directive  80-6. 

Lessons  Learned  Staff  Officer  Training 

You  should  attend  the  week-long  NATO  LLSO  Course.  Courses  are  offered  four  times 
a  year.  More  information  can  be  found  via  the  SWEDINT  website3.  Individual  nations 
may  also  offer  LL  training  as  part  of  their  national  training  programmes. 

Using  This  NATO  Lessons  Learned  Handbook 

This  handbook  is  divided  into  seven  chapters  advising  you  on  various  aspects  of  LL 
and  contains  annexes  with  helpful  examples  and  templates. 

•  Chapter  1  is  this  introduction  which  provides  an  overview  of  LL. 

•  Chapters  2-5  describe  the  LL  process: 

•  Chapter  2  provides  an  overview  of  an  LL  process  based  on  the  NATO  LL  Policy, 
gives  definitions,  and  outlines  some  of  the  actors  involved  in  the  NATO  LL 
process. 

•  Chapter  3  gives  guidance  on  gathering  observations  and  introduces  a  template 
forgathering  and  documenting  lessons. 


3  http://www.forsvarsmakten.se/en/Organisation/Centres/Swedish-Armed-Forces-lnternational- 
Centre/Courses-at-SWEDINT/NATO-LL-SOC/ 
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•  Chapter  4  guides  you  on  the  process  of  analysis  that  facilitates  development  of 
observations  into  useful  lessons. 

•  Chapter  5  offers  practical  advice  on  management  of  the  remedial  action  process 
to  ensure  lessons  are  truly  learned. 

•  Chapter  6  describes  LL  Information  Sharing  and  the  LL  Community. 

•  Chapter  7  gives  a  summary  of  this  handbook  and  where  to  go  for  more 
information. 

•  References 

•  Annex  A:  Lessons  Learned  Glossary 

•  Annex  B:  Observation,  Discussion,  Conclusion,  Recommendation  Template 

•  Annex  C:  Example  lessons  learned  in  the  observation,  discussion,  conclusion 
and  recommendation  lesson  template  that  is  advocated  by  this  handbook 

•  Annex  D:  Interview  Process 

•  Annex  E:  LL  Capability  Checklist 

We  advise  you  to  read  all  chapters  quickly  and  then  go  back  through  and  make  notes 
regarding  how  the  concepts  presented  relate  to  your  role  and  your  organization 
specifically.  The  annexes  can  be  used  to  increase  your  understanding  and  support 
your  daily  work  as  appropriate.  Each  chapter  has  some  introductory  remarks  and 
concludes  with  a  summary  of  the  chapter  contents  in  bullet  form.  Boxes  are  used  to 
highlight  additional  information  about  particular  subjects  and  make  tips  stand  out. 

JALLC  Advisory  and  Training  Team 

The  JALLC  Advisory  and  Training  Team  (JATT)  is  tasked  with  LL  outreach.  The  JATT 
can  provide  advice,  assistance  and  training  to  NATO  commands,  HQs,  and  Alliance 
and  partner  nations  upon  request  to  aid  development  of  LL  capabilities.  JATT  can  be 
reached  via  e-mail  at  iattpoc@iallc.nato.int. 


SUMMARY 

Lessons  Learned 

•  Lessons  Learned  describes  activities  relating  to  learning  from  experience  to  achieve 
improvements.  In  a  military  context,  this  means  reduced  operational  risk,  money 
saved,  and  improved  operational  effectiveness. 

•  Lessons  can  be  derived  from  any  activity  -  daily  events,  exercises,  training,  etc. 

•  Learning,  in  any  organization,  involves  three  generic  stages:  identification,  action, 
and  institutionalization. 

•  A  NATO  Lesson  Learned  results  from  the  implementation  of  a  remedial  action  that 
produced  an  improved  performance  or  increased  capability. 

Role  of  a  Lessons  Learned  Staff  Officer 

•  Support  the  LL  Process  -  gather,  staff,  action  and  communicate  lessons  to  ensure 
learning  from  the  experience  is  converted  into  actual  improvement. 

•  Support  LL  Information  Sharing  -  share  lessons  both  within  and  outside  of  the 
organization  via  databases,  websites,  reports,  newsletters,  etc. 

•  Support  the  LL  Community  -  attend  and  organize  relevant  LL  sharing  events  (LL 
conference,  forums,  working  groups,  etc.). 

•  Support  LL  Capability  -  Set  up  or  improve  the  organization’s  LL  capability. 

Getting  Started 

•  Review  NATO  LL  references  and  study  this  handbook 

•  Request  JATT  outreach  support  and  attend  an  LLSO  Course.. 
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NATO  LL  Process  and  Actors 

The  most  formal  approach  to  LL  is  the  use  of  an  LL  process.  A  LL  process  is  a 
procedure  for  deliberately  staffing  observations  arising  from  an  activity  until  a  lesson 
learned  is  reached. 

This  chapter  provides  an  overview  of: 

•  NATO  Lessons  Learned  Process 

•  NATO  Actors  In  The  Lesson  Learned  Process 

NATO  LESSONS  LEARNED  PROCESS 

Figure  1  illustrates  the  LL  process  used  by  NATO  (Reference  A).  The  text  that  follows 
describes  the  process.  Terms  with  specific  NATO  definitions  are  defined  as  they  are 
used  and  their  definitions  are  included  in  an  LL  Glossary  at  Annex  A. 


Figure  1:  The  NATO  LL  process 

Note  that  this  process  follows  the  three  basic  generic  stages  of  learning  described  in 
Chapter  1 .  ‘Identification’  occurs  during  the  Observation  to  LI  part  of  the  process  and 
‘Action’  and  ‘Institutionalization’  occur  during  the  LI  to  Lesson  Learned  part  of  the 
process.  In  NATO,  ‘Institutionalization’  is  seen  as  an  integral  part  of  the  action 
necessary  to  reach  a  Lesson  Learned.  Information  generated  during  the  LL  process 
can  be  shared  at  any  time.  More  information  on  LL  Information  Sharing  is  provided  in 
Chapter  6  of  this  handbook. 
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The  LL  process  starts  with  Gathering  Observations. 

•  An  Observation  is  “an  issue  identified  for  improvement”  (Reference  C). 

For  a  given  activity,  an  expected  outcome  exists.  If  everything  goes  as  expected,  there 
is  little  to  learn  from  the  activity.  However,  if  expectations  are  either  not  met  or 
exceeded,  there  is  something  to  learn.  Any  difference  from  expected  outcome  should 
be  documented  as  an  observation  that  describes:  the  sequence  of  events,  conditions 
under  which  the  events  occurred,  and  other  quantifying  details.  Observations  are 
further  discussed  in  Chapter  3. 

The  observer  may  conduct  some  initial  analysis  to  elaborate  on  the  reason(s)  why  the 
activity  differed  from  expectation  and  identify  a  proposed  solution.  For  more  complex 
observations,  further  analysis  may  be  required. 

•  Analysis  is  “the  study  of  a  whole  by  thoroughly  examining  its  parts  and  their 
interactions”4. 

Analysis  will  be  used  to  find  the  root  cause(s)  of  the  observed  issue  and  identify  a 
Remedial  Action  (RA). 

•  An  RA  is  “a  possible  action  that  serves  to  rectify  a  fault  or  improve  conditions” 
(Reference  C). 

Additionally,  the  person  or  organization  which  should  execute  the  RA  will  be  identified 
during  the  Analysis  step.  The  aim  of  the  analysis  is  to  generate  a  LI. 

•  A  Lesson  Identified  (LI)  is  “an  observation  for  which  an  RA  has  been 
developed  and  an  Action  Body  (AB)  to  carry  out  the  RA  has  been  proposed” 
(Reference  C). 

Analysis  is  further  described  in  Chapter  4. 

Tip:  Analysis 

Do  not  get  too  worried  about  the  use  of  the  term  “analysis”.  Analysis  may  convey  the  impression 
that  some  mysterious,  formal  intellectual  activity  must  take  place  to  be  able  to  develop  a  LI.  In 
many  cases,  a  healthy  dose  of  common  sense  and  SME  Input  is  sufficient  to  determine  root 
causes  and  identify  appropriate  actions. 

A  special  type  of  LI  is  a  Best  Practice  (BP). 

•  A  BP  is  “an  activity  ora  series  of  activities  proven  effective  through  analysis  that 
can  be  replicated  by  others  in  a  similar  situation”  (Reference  D). 

When  an  LI  is  a  BP,  the  RA  will  be  to  document  the  conditions  under  which  the  positive 
experience  occurred  and  introduce  measures  to  ensure  these  conditions  are  repeated. 
BPs  may  be  specific  to  an  environment,  theatre,  or  situation,  and  may  become 
obsolete.  A  BP  may  require  validation,  and  should  be  regularly  reviewed  to  ensure  that 
the  practice  is  still  “best”. 

The  next  step  in  the  LL  process  is  Endorsement  and  Tasking.  During  this  step, 
developed  Lis  will  be  presented  to  the  organization’s  leadership  for  them  to  determine 
how  to  progress  the  LI  through  the  LL  process.  First  the  LI  will  be  endorsed  whereby  it 
is  approved  for  further  action  and  the  proposed  RA  is  accepted  or  modified  to  be 
acceptable,  then  an  AB  will  be  tasked  to  plan  and  implement  the  RA. 

•  An  AB  is  “an  organization  tasked  to  implement  or  facilitate  the  implementation  of 
an  approved  action”  (Reference  C). 

During  the  Implementation  and  Monitoring  step,  the  AB  will  implement  their  RA  plan 
and  the  LLSO  will  support  leadership  in  monitoring  its  implementation. 


4  AAP6  NATO  Glossary  of  Terms  and  Definitions,  NATO  Standardization  Agency,  2010. 
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After  the  RA  has  been  implemented,  validation  will  be  completed. 

•  Validation  “determines  if  the  issue  that  was  originally  observed  has  been 
successfully  rectified  by  the  RA  carried  out  in  accordance  with  the  action  plan”5 
(Reference  C). 

Validation  may  involve  exercises  or  experiments.  Following  the  completion  of  the  RA 
and  successful  validation,  the  LI  will  be  deemed  a  Lesson  Learned  and  the  LL  process 
concludes.  The  last  three  steps  of  the  LL  process  are  described  in  Chapter  5. 

Sharing  is  an  activity  that  needs  to  occur  through  the  LL  process.  More  about  LL 
information  sharing  is  in  Chapter  6. 

NATO  ACTORS  IN  THE  LESSON  LEARNED  PROCESS 

NATO  HQ  /  NATO  Agencies  Lesson  Learned  Points  of  Contact  Network 

The  NATO  HQ  /  NATO  Agencies  LL  POC  network  is  the  focus  for  LL  in  NATO  HQ  and 
the  Agencies  as  described  in  the  information  memo  subject  NATO  HQ  /  NATO 
Agencies  Lessons  Learned  Process  -  Implementation6.  It  meets  bi-monthly  and  is 
chaired  jointly  by  the  International  Staff  (IS)  and  the  International  Military  Staff  (IMS) 
representatives.  It  comprises  representatives  from  all  NATO  agencies,  NATO  HQ 
(IS/IMS)  Divisions  and  NATO  Military  Authorities.  The  LL  POC  network  will  ensure  a 
coordinated  and  consistent  approach  within  NATO  HQ/Agencies  and  will  also  monitor 
progress  of  the  LL  process  implementation.  The  LL  POC  network  is  responsible  for 
identifying  trends  in  internal  lessons. 

Military  Committee  (MC)  Standardization  Boards 

The  MC  delegates  tasking  authorities  for  operational  standardization,  including  NATO 
doctrine,  to  Standardization  Boards.  MC  Joint,  Maritime,  Land,  Air  and  Medical 
Standardization  Boards  with  their  respective  Working  Groups  are  responsible  for  the 
development  of  operational  and  procedural  standardization.  They  produce  NATO 
Standardization  Agreements  and  Allied  Publications  with  the  member  countries  and 
NATO  Military  Commands  and  agencies.  The  Standardization  Boards  are  supported 
by  the  NATO  Standardization  Agency.  Additionally,  the  Standardization  Boards  and 
their  respective  Working  Groups  can  provide  subject  matter  expertise  to  review  lessons 
related  to  NATO  doctrine  and  procedures  as  promulgated  in  the  Allied  Publications. 

Conference  of  National  Armaments  Directors  Groups 

The  Conference  of  National  Armaments  Directors  (CNAD)  and  its  subordinate 
structures  are  focused  on  the  cost-effective  acquisition  by  Alliance  nations  of  military 
capabilities,  by  enhancing  and  encouraging  interoperability  and  promoting 
technological  cooperation.  The  CNAD  groups  include: 

•  NATO  Naval  Armaments  Group 

•  NATO  Air  Force  Armaments  Group 

•  NATO  Army  Armaments  Group 


5  Note  that  within  the  NATO  LL  process,  the  term  validation  is  applied  to  the  confirmation  that  a 
remedial  action  is  successful  at  fixing  the  observed  problem.  Other  LL  processes  may  use  the 
term  differently;  it  is  also  used  by  some  to  describe  the  process  to  determine  whether  an 
observation  is  suitable  for  inclusion  in  the  LL  process. 

6  NATO  International  Staff,  Information  Memo  Subject:  NATO  HQ/  NATO  Agencies  Lessons 
Learned  Process  -  Implementation,  3  July  2009,  NATO  Unclassified. 
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The  NATO  Division  of  Defence  Investment  Armaments  Directorate  supports  the  work 
of  the  CNAD  and  its  groups.  The  groups  are  good  POCs  within  NATO  for  lessons 
regarding  capabilities  and  interoperability  of  equipment. 

Supreme  Allied  Commander  Transformation  (SACT)  and  Supreme  Allied 
Commander  Europe  (SACEUR) 

The  Bi-SC  Directive  80-6  LL  (Reference  C)  establishes  the  SACT  as  lead  for  the 
overall  NATO  LL  process  for  lessons  at  the  strategic  command  level  and  above.  SACT 
is  supported  by  SACEUR  in  the  planning  and  execution  of  this  task.  Reciprocally, 
SACT  supports  SACEUR  with  ensuring  the  overall  output  of  the  NATO  LL  process  is 
made  available  for  the  planning  and  execution  of  operations,  exercises  and  training. 

HQ  SACT 

The  focal  point  at  HQ  SACT  for  all  matters  relating  to  LL  is  the  LL  Implementation 
Branch  under  Assistant  Chief  of  Staff  Programme  and  Planning  Management.  The  LL 
Implementation  Branch  represents  ACT  at  the  Bi-SC  LL  Steering  Group.  HQ  SACT 
tasks  JALLC  to  conduct  LL  analysis  projects  on  an  annual  basis  and  with  emergent 
analysis  requirements  for  immediate  execution  as  necessary. 

SHAPE 

The  focal  point  at  SHAPE  for  all  matters  relating  to  LL  is  the  Force  Standards  and  LL 
Branch  in  the  Readiness  and  Requirements  Directorate.  This  branch  coordinates  the 
internal  staffing  of  LL  reports  and  proposes  AB  tasking  to  SHAPE  Chief  of  Staff.  RAs 
that  exceed  SHAPE’S  capacity  or  scope  will  be  forwarded  to  NATO  HQ  or  HQ  SACT. 
SHAPE  Force  Standards  and  LL  Branch  represents  ACO  in  the  Bi-SC  LL  Steering 
Group.  SHAPE  also  receives  and  reviews  lessons  forwarded  by  the  Joint  Force 
Commands  (JFCs). 

Joint  Force  Commands  and  Component  Commands 

The  JFCs  and  Component  Commands  (CCs)  create  and  maintain  their  own  LL 
directives  and  SOPs.  When  a  LI  RA  falls  under  their  remit,  the  JFC/CC  endorses  the 
LI  and  develops  an  internal  action  plan  to  remedy  deficiencies.  The  JFC/CC 
implements  RAs  as  tasked  by  SHAPE.  Biannually,  JFCs/CCs  report  LI  to  SHAPE  in 
the  lesson  template  format  described  in  Chapter  3  and  Annex  B.  Lessons  reported  to 
SHAPE  include  those  with  potential  applicability  to  other  commands  or  organizations, 
and  those  where  the  RA  is  beyond  the  capacity  or  scope  of  the  JFC/CC.  If  a  LI  is 
determined  to  need  urgent  attention,  the  JFC/CC  immediately  reports  the  LI  to  SHAPE 
for  action.  (Reference  D) 

JALLC 

The  JALLC  is  NATO’s  centre  for  performing  joint  analysis  of  operations,  training, 
exercises  and  concept  development  and  experimentation  collective  experiments.  It 
has  established  and  maintains  the  NATO  LL  Database  (LLDb)  and  is  the  focal  point  for 
analysis  of  LL7.  JALLC  facilitates  the  sharing  of  LL  among  Allies  as  well  as  with  non- 
NATO  nations  and  international  organizations  as  appropriate.  On  request,  JALLC  can 
engage  with  nations  and  NATO  commands  and  agencies  to  support  the  development 
of  their  LL  capability  (Reference  A). 


7  MC  510(FINAL),  Terms  of  Reference  for  Directors  JWC,  JFTC  and  JALLC,  26  April  2004, 
NATO  Restricted. 
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Centres  of  Excellence 

Centres  of  Excellence  provide  subject  matter  expertise  to  assist  in  the  analysis  of 
issues.  They  also  represent  potential  communities  of  interest  (see  Chapter  6)  to  share 
lessons  on  particular  subjects. 

Commanders 

Commanders  at  all  levels  are  responsible  for  their  LL  processes  in  accordance  with  the 
NATO  LL  Policy  (Reference  A)  and  the  Bi-SC  Directive  80-6  LL  (Reference  C).  LL  is  a 
command  responsibility  and  each  command  under  the  strategic  command  structures 
will  operate  its  own  internal  LL  processes. 


SUMMARY 

NATO  Lessons  Learned  Process 

•  Initially,  differences  between  expectations  and  actual  performance  are  identified. 
These  observations  are  the  starting  block  for  the  process. 

•  Analysis  of  observations  identifies  root  causes,  remedial  actions,  and  the 
appropriate  action  body  to  execute  the  action.  These  items  together  form  a  LI. 

•  Leadership  will  review  Lis  to  determine  how  to  proceed  with  the  LL  process. 
Endorsement  and  tasking  as  well  as  implementation  and  monitoring  are  a 
leadership  responsibility. 

•  Upon  successful  validation,  a  LI  becomes  a  Lesson  Learned. 

NATO  Actors  in  the  LL  Process 

•  LL  is  a  command  responsibility  and  each  command  under  the  strategic 
command  structures  will  operate  its  own  internal  LL  processes. 

•  The  many  different  actors  within  the  process  will  take  on  different  roles 
depending  on  the  scope  of  the  lesson.  The  chain  of  command  must  be  set  up  to 
ensure  lessons  are  worked  through  the  LL  process. 

•  At  times,  the  different  actors  may  provide  subject  matter  expertise  to  a  process 
or  simply  be  a  member  of  the  community  of  interest. 
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Gathering  Observations 

An  observation  is  the  basis  of  an  LL  process.  The  observation  must  convey  the  basic 
details  of  the  observed  issue,  with  detail  sufficient  for  further  analysis.  At  a  minimum, 
the  observation  must  address  the  questions  “what  happened?”  and  “how  did  that  differ 
from  what  was  expected?”  One  of  the  LLSOs  most  important  tasks  is  gathering 
observations  from  the  organization  as  they  occur,  especially  following  planned  events 
or  activities.  Once  gathered,  observations  should  be  reviewed  to  filter  out  unsuitable 
observations  (such  as  certain  obvious  complaints),  either  by  an  officer  higher  in  the 
immediate  chain  of  command  or  by  a  dedicated  LL  manager  (such  as  the  LLSO). 
Observations  that  survive  this  initial  review  process  are  deemed  suitable  for  inclusion  in 
the  LL  process  and  will  need  to  be  managed.  This  chapter  provides  general  guidance 
on  the  different  aspects  of  gathering  observations  including: 

•  Capturing  Observations 

•  Managing  Observations 

•  Tools  to  Capture  and  Manage  Observations 

CAPTURING  OBSERVATIONS 

All  personnel,  regardless  of  rank,  must  understand  they  have  a  responsibility  to 
document  observed  problems,  shortfalls  and  successes.  There  must  be  provisions  for 
all  to  make  observations.  Ideally,  observation  reporting  will  complement  routine  reports 
and  returns,  not  replace  them.  These  observations  from  within  the  organization  are 
likely  to  be  the  main  focus  for  observation  capture.  However,  observations  from 
outside  the  organization  should  also  be  captured  and  reviewed. 

Observation  Attribution 

To  encourage  the  reporting  of  negative  experience,  some  organizations  allow 
observers  to  remain  anonymous.  Although  this  does  encourage  reporting,  it  is  not  a 
good  practice  in  the  context  of  an  LL  process.  Where  observations  are  drafted  with 
insufficient  detail  by  anonymous  originators  you  may  not  be  able  to  obtain  further 
information.  A  compromise  should  be  reached  whereby  observations  can  be  submitted 
with  a  degree  of  anonymity  for  the  observer  while  allowing  you  to  find  out  further 
information  if  necessary.  This  could  be  achieved  by  attributing  the  observation  to  the 
branch  or  unit. 

Observation  Template 

There  are  long-term  benefits  to  conforming  to  the  NATO-wide  lesson  template 
described  below  for  capturing  observations.  Use  of  a  common  template  allows 
information  to  be  shared  more  easily.  Where  the  template  for  observations  differs,  the 
automatic  transfer  of  data  can  be  difficult. 

The  template  for  a  lesson  suggested  in  ACO  Directive  80-1  Lessons  Learned 
(Reference  D)  contains  five  fields:  Title,  Observation,  Discussion,  Conclusion,  and 
Recommendation  (ODOR).  This  is  the  format  that  entries  into  the  NATO  LLDb  must 
take  in  accordance  with  Bi-SC  Lessons  Learned  Directive  80-6  (Reference  C)  and 
therefore  it  is  recommended  that  you  collect  observations  in  this  template  from  the 
start. 

Annex  B  explains  the  template  in  full  and  Annex  C  provides  Joint,  Maritime  and  Land 
examples  using  the  template. 
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Post  Event  Reporting 

Post  event  reports  are  an  ideal  means  to  capture  observations.  The  Bi-SC  Exercise 
Directive  75-3  (Reference  E),  supplemented  by  individual  sub-command  directives  and 
SOPs,  gives  specific  direction  on  the  format  and  timeline  for  production  of  post-event 
reports,  including  interim  reports.  This  reporting  should  already  contain  much  of  the 
information  you  need  to  capture  observations.  In  general,  some  important  factors  in 
post-event  information  capture  include: 

•  Capturing  adequate  data  in  a  timely  manner. 

•  Capturing  the  data  in  a  common  format. 

•  Applying  quality  analysis  to  the  data. 

•  Prioritizing  issues. 

•  Producing  information  that  can  be  shared  with  the  appropriate  community  of 
interest. 

•  Ensuring  the  final  product  is  focused  on  guidance  for  the  next  event’s  planners, 
information  for  the  previous  event’s  players  to  review. 

Tip:  Focus  of  Post  Event  Reporting 

The  key  consideration  in  post-event  reporting  is  ensuring  the  final  product  is  focused  on  giving 
guidance  to  the  next  event’s  planners,  not  the  previous  event’s  players  for  review.  This  should 
be  the  focus  to  ensure  that  mistakes  are  not  repeated  and  best  practice  is  institutionalized. 

Without  this  focus,  post-event  products  tend  to  be  shelved  and  do  not  become  part  of  the 
knowledge  base  of  the  next  rotation  of  actors  in  the  exercise  or  operation. 


Post-Operational  and  Post-Exercise  Interviews 

One  way  to  convert  tacit  knowledge  gained  during  operations  and  exercises  into 
explicit  knowledge  for  inclusion  in  the  observation  is  by  interviewing  SMEs.  Post¬ 
operation  or  post-exercise  interviews  are  a  valuable  way  of  capturing  lessons  before 
troops  and  key  leaders  disperse,  and  while  the  memories  of  the  events  are  still  clear. 
Interviews  have  the  added  advantage  of  allowing  the  interviewer  to  focus  on  interest 
areas  that  the  interviewee  may  otherwise  not  consider  as  important. 

The  US  Center  for  Army  Lessons  Learned  (CALL)  coordinates  what  is  known  as  an 
“Umbrella  Week”  with  forces  returning  to  home  units,  prior  to  post-deployment  leave. 
During  this  week,  CALL  interviews  many  of  the  troops  to  capture  observations.  This  is 
the  only  opportunity  that  CALL  has  to  extract  potential  lessons  from  returning  troops 
before  they  disperse. 

Annex  D  outlines  a  basic  interview  process  which  allows  the  interviewer  to  methodically 
obtain  a  broad  outline  of  events,  while  focusing  on  key  events  and  their  causal  factors. 

MANAGING  OBSERVATIONS 

Observations  will  arrive  in  many  different  formats,  levels  of  maturity  and  from  many 
different  sources.  You  will  need  to  review  observations  for  maturity  and  suitability,  and 
ensure  observations  are  stored  with  appropriate  metadata  to  comply  with  information 
management  best  practices. 

Reviewing  Observations 

Observations  should  be  reviewed  on  the  basis  of  suitability  for  inclusion  in  an  LL 
process  as  soon  as  possible  after  capture.  This  initial  review  process  can  be  carried 
out  either  by  a  member  of  the  originator’s  chain  of  command  or  by  a  dedicated  LL 
manager  (such  as  the  LLSO).  This  initial  review  process  acts  as  a  filter  to  remove 
unsuitable  observations.  When  carried  out  immediately  after  the  observation  has  been 
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captured,  the  reviewing  officer  will  be  able  to  get  back  to  the  originator  to  clarify  any 
points  or  issues  while  it  is  still  fresh  in  the  originator’s  mind. 

For  this  initial  review,  ask  the  following  questions8 9 10: 

1.  Is  this  an  objective  observation  and  not  just  an  obvious  complaint  about 
something  or  somebody? 

2.  Is  this  a  problem  with  the  system  and  not  just  a  simple  mistake  by  somebody? 

3.  Does  this  adequately  and  correctly  describe  the  observed  situation? 

4.  Would  you  spend  your  own  money  to  fix  this  issue? 

5.  Would  you  spend  your  own  time  fixing  this  issue? 

The  answer  to  all  of  these  questions  should  be  yes  for  an  observation  to  be  suitable  for 
inclusion  in  the  LL  process.  Inclusion  means  somebody  will  spend  some  sort  of 
resources  to  address  the  issue  contained  in  the  observation.  Observations  for 
inclusion  pass  into  the  next  stage  of  the  LL  process  where  the  first  task  is  to  review  its 
maturity  and  judge  whether  it  is  already  a  LI  or  whether  it  needs  further  analysis  to 
become  an  LI  (see  Chapter  4). 

If  you  decide  an  observation  is  unsuitable  for  inclusion  in  the  LL  process,  it  can  either 
be  deleted  or  placed  into  an  archive.  You  should  inform  the  originator  of  the  decision 
that  has  been  made  regarding  their  observation  so  that  they  know  their  observation  has 
been  considered  and  not  just  forgotten. 

Observation  Metadata 

You  should  attach  metadata  to  observations  captured  as  soon 
as  possible  in  accordance  with  NATO  Primary  Directive  on 
Information  Management  (Reference  F).  This  will  aid  future 
management  of  information  and  facilitate  information  retrieval 
and  sharing. 

Consider  carefully  what  metadata  you  want  to  attach  to  your  observations;  it  will  save  a 
lot  of  time  in  the  long  term  if  you  get  it  right  the  first  time.  The  metadata  you  attach 
should  be  NATO  UNCLASSIFIED  or  non-classified,  even  if  the  observation  it  is 
describing  is  of  a  higher  classification.  Keeping  the  metadata  unclassified  will  help 
later  on  with  LL  information  sharing.  As  a  minimum,  the  following  set  of  metadata  is 
required  in  accordance  with  NATO  Primary  Directive  on  Information  Management 
(Reference  F): 

•  Originator:  The  person  or  organization  that  initially  identified  the  observation. 
Ideally  this  will  be  a  person  but  at  least  the  originating  branch  or  unit  is  needed. 

•  POC:  The  person  or  branch  that  will  manage  the  information  after  it  has  been 
submitted.  This  will  usually  be  the  LLSO. 

•  Classification:  An  appropriate  classification  for  the  observation.  Give  some 
thought  to  the  classification  of  the  observation.  Ensure  compliance  with  security 
guidelines9,10  but  resist  the  temptation  to  over-classify:  it  may  prevent  you  from 


Definition:  Metadata 

Noun,  a  set  of  data  that 
describes  and  gives 
information  about  other 
data. 


8  By  posing  and  answering  these  questions,  you  are  already  performing  an  initial  analysis  of  the 
observation.  Trust  your  instincts;  if  it  smells,  there  is  probably  a  reason  for  that. 

9  NATO  Security  Committee,  Guidelines  on  the  Security  of  Information,  May  2005,  AC/35- 
D/1032,  NATO  Unclassified. 

10  North  Atlantic  Council,  The  Management  of  Non-Classified  NATO  Information,  23  July  2002, 
C-M(2002)60,  NATO  Unclassified. 
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sharing.  If  the  originator  has  already  added  a  classification,  review  it.  If  it  seems 
that  the  classification  is  inappropriate,  go  back  to  the  originator  to  get  it  changed. 

•  Releasability:  An  appropriate  releasability  to  allow  for  the  widest  reasonable 
distribution.  Again,  think  carefully,  in  particular  about  future  opportunities  to 
share  the  information  contained  in  the  observation.  As  with  classification,  the 
originator  will  need  to  approve  any  changes  to  the  releasability  that  are 
necessary. 

•  Date:  The  date  the  observation  was  identified.  This  will  allow  people  to  know 
how  old  the  information  is  and  to  judge  whether  it  is  still  current. 

•  Title:  A  statement  that  encapsulates  the  essence  of  the  subject  of  the 
observation  or  lesson  in  such  a  way  to  give  a  reasonable  indication  as  to  the 
content.  If  the  lesson  template  (Annex  B)  is  used,  this  will  be  the  same  as  the 
title  for  the  observation. 

Here  are  some  ideas  that  you  may  consider  for  other  useful  metadata: 

•  Time 

•  Place,  position  or  location 

•  Name  of  operation,  exercise  or  experiment 

•  Priority 

•  Impact:  Mission  critical,  mission  desirable,  mission  useful 

•  (Expected)  Frequency  of  occurrence:  Frequent,  Occasional,  Rare 

•  Relevant  NATO  Task  List  item 

•  Essential  Operational  Capabilities 

•  Levels:  Political,  Strategic,  Operational,  Tactical 

•  NATO  lines  of  capability  development:  Doctrine,  Organization,  Training, 

Materiel,  Leadership,  Personnel,  Facilities  -  Interoperability  (DOTMLPF-I) 

•  Source:  Direct  observation,  interview,  instrumentation  data,  survey. 

TOOLS  TO  CAPTURE  AND  MANAGE  OBSERVATIONS 

A  tool  should  be  used  to  support  the  collection  of  observations  to  ensure  that 
observations  can  be  collated,  processed,  prioritized,  and  shared.  The  tool  used  should 
be  as  simple  as  possible  and  should  complement  your  organization’s  procedures  for 
processing  and  sharing  information.  Some  units  favour  a  notebook  for  recording 
observations.  This  approach  is  simple  and  inexpensive,  but  software  solutions  offer 
alternatives  that  can  make  tracking,  processing  and  sharing  observations  easier. 

Web-based  Systems 

Microsoft  SharePoint  Server  and  similar  systems  are  web-based  content  manager 
systems.  The  use  of  SharePoint ,  when  and  where  available,  is  probably  the  best  way 
to  collect  observations  because  it  has  webparts  which  make  it  easy  to  create  a  simple 
form  for  collecting  observations  that  can  be  used  over  an  internal  network  or  the 
internet.  Observers  can  simply  click  on  a  link,  enter  the  data,  and  submit  the  form. 
Observation  records  will  be  automatically  time  stamped  and  tagged  with  the  originator’s 
log-in  name.  SharePoint  can  automatically  export  the  submitted  observation  to 
Microsoft  Excel  or  Microsoft  Access  for  further  processing.  The  NATO  LL  Portal, 
accessible  via  the  JALLC  website,  is  based  on  SharePoint  technology. 
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Observation  Collection  Programme 

NATO’s  Observation  Collection  Program  (OOP)  is  a  Microsoft  Windows  application  that 
allows  observations  to  be  entered  in  a  systematic  and  easily-recoverable  manner  by 
individuals  or  teams.  The  software  is  freely  available  on  the  internet  via 
http://www.iallc.nato.int/Documents/  or  on  the  NATO  Secret  network.  OOP  allows 
observers  to  register  free  text  observations  and  associated  discussions,  conclusions 
and  recommendations,  categorize  these  observations  with 
respect  to  lists,  and  exchange  information  with  other  observers. 

Additional  guidance  may  be  found  in  the  OCP  Quick  Start 
Guide,  the  OCP  Administration  Manual  and  the  OCP  User’s 
Manual,  and  directly  from  IT  support  at  the  JALLC  at 
tlcqcx0010@iallc.nato.int. 

Microsoft  Office  Software 

Microsoft  Word,  Excel  and  Access  are  simple  and  widely  available  tools  that  can  be 
used  to  store  and  manage  observations.  Many  users  will  be  familiar  with  these  tools 
and  will  have  the  software  installed  on  their  computer,  encouraging  easy  sharing.  Ease 
of  use  and  familiarity  are  important  considerations  in  encouraging  people  to  submit 
observations.  A  short  overview  of  the  advantages  and  disadvantages  Word,  Excel  and 
Access  have  for  supporting  observation  capture  follows: 

Microsoft  Word 

Advantages:  Familiarity;  ease  of  use;  ease  of  setting  up;  ability  to  store  metadata  in 
file  properties;  ease  of  sharing. 

Disadvantages:  Difficult  to  manage  many  observations;  no  filtering  of  observations; 
limited  sorting  of  observations;  limited  search  capabilities;  poor  data  integrity 
protection. 

Microsoft  Excel 

The  JALLC  has  produced  a  Microsoft  Excel  workbook  that  uses  a  number  of  macros  to 
facilitate  the  entering,  editing  and  management  of  observations  that  is  available  for  use 
by  the  NATO  LL  community  via  the  JALLC  website11. 

Advantages:  Familiarity;  ease  of  use;  ease  of  setting 
up;  easy  metadata  tagging;  powerful  filtering,  sorting 
tools;  good  search  capabilities;  easy  to  share. 

Disadvantages:  Merging  independent  data  files  is 
difficult;  only  1024  characters  display  in  a  cell  (in  older 
versions);  relatively  poor  data  integrity  protection  (easy 
to  delete  and  edit  entries  by  accident). 

Microsoft  Access 

Advantages:  A  relational  database  can  store  lots  of  data  very  efficiently;  excellent 
browsing,  filtering,  sorting  and  custom  reporting  capabilities;  good  data  integrity 
protection. 

Disadvantages:  A  relational  database  can  be  very  complex  to  set  up  and  maintain; 
majority  of  users  will  be  less  familiar  with  Microsoft  Access  than  with  other  Microsoft 
Office  applications;  Microsoft  Access  is  not  a  component  of  the  some  Microsoft  Office 
installations. 


Tip:  Using  Excel  to  Gather 

Observations 

•  Use  columns  for  ODCR 
fields,  rows  for  individual 
observations. 

•  Check  out  AutoFilter  for 
filtering  and  searching 
(select  the  “ contains ” 
keyword  in  the  custom 
AutoFilter). 


Tip:  OCP 

On  installing  OCP, 
there  is  a  default 
ADMIN  account  (no 
password  required). 


11  http://www.jallc.nato.int 
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The  NATO  Lessons  Learned  Database 

Many  headquarters  use  tailor-made  LLDbs  to  collect  and  store  observations.  The 
NATO  LLDb  is  a  tailor-made  database  that  can  be  used  to  collect  observations  and 
facilitate  their  browsing,  searching,  filtering,  sorting,  reporting  and  archiving.  However, 
it  was  primarily  designed  as  a  tool  to  support  the  staffing  of  lessons  through  an  LL 
process,  and  in  particular  the  NATO  Bi-SC  LL  process.  Therefore,  it  may  not  be  the 
most  appropriate  tool  simply  for  storing  and  managing  observations.  The  use  of  the 
NATO  LLDb  to  support  the  staffing  of  lessons  is  discussed  in  Chapter  5. 

Tip:  Software  Tools  for  Observation  Collection 

When  considering  software  tools  to  support  observation  collection,  the  following  questions 
should  be  addressed,  in  addition  to  the  usual  cost  and  maintenance  considerations: 

□  Is  the  software  easy  to  use  and  familiar  to  users? 

□  How  will  the  observation  collection  capability  be  deployed:  stand-alone  PCs;  over  a  local 
area  network;  over  a  wide-area  network;  over  the  internet? 

□  Will  files  need  to  be  centrally  accessed  or  circulated? 

□  What  are  the  bandwidth  requirements  and  file  sizes? 

□  Will  the  information  remain  current?  How  will  versions  be  controlled? 

□  What  contributing  and  editing  rights  and  limitations  are  required? 

□  What  browsing,  filtering  and  sorting  capabilities  are  needed? 

□  Can  searches  be  performed  adequately? 

□  Can  the  information  be  updated  easily? 

□  Can  supporting  information  such  as  images  be  attached? 

□  What  report  generation  capability  is  needed? 

□  What  staffing  processes  will  need  to  be  supported  by  the  software? 


SUMMARY 

Capturing  Observations 

•  Provisions  should  be  in  place  for  all  personnel  regardless  of  rank  to  document 
observed  problems,  shortfalls  and  successes. 

•  Ideally,  the  person  reporting  an  observation  will  attach  their  name  to  the  observation. 
However,  if  a  degree  of  anonymity  is  desired,  provisions  must  be  in  place  to  identify 
the  observation  with  the  appropriate  branch  or  unit  to  allow  a  clear  basis  for  further 
staffing  if  required. 

•  Using  the  ODOR  template  to  record  observations  enhances  interoperability  by 
allowing  information  to  be  shared. 

•  Post-event  reports  are  an  ideal  source  for  observations  and  should  become  a  part  of 
the  knowledge  base  for  the  next  event’s  planners  to  use. 

•  Post-event  interviews  are  a  valuable  way  to  capture  lessons. 

Managing  Observations 

•  Observations  should  be  reviewed  as  soon  as  possible  after  capture  to  filter  out 
unsuitable  observations  and  allow  for  the  capture  of  additional  information. 

•  From  the  start  of  the  process,  attach  metadata  to  the  observations.  Metadata  will 
make  finding  and  subsequently  sharing  information  easier.  Careful  consideration 
should  be  given  to  the  metadata  used  and  should  be  NATO  UNCLASSIFED  or  non- 
classified. 

Tools  to  Capture  and  Manage  Observations 

•  Methods  for  collecting  observations  should  be  as  simple  as  possible  and  should 
complement  procedures  for  processing  and  sharing  lessons. 

•  Software  solutions  can  make  tracking,  processing  and  sharing  observations  easier. 
Whichever  tool  is  used  should  be  one  that  all  users  can  and  will  use. 
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4 

Analysis  -  Observations  to  Lessons  Identified 

Once  it  is  decided  that  an  observation  is  suitable  for  inclusion  in  the  LL  process,  the 
next  stage  is  its  transition  from  an  observation  to  a  LI.  Analysis  is  generally  completed 
in  two  stages:  first  to  find  root  cause(s)  and  second  to  determine  Remedial  Action(s) 
(RA).  This  chapter  will  explore  methods  of  analysis  and  walk  you  through  the  transition 
of  an  observation  to  a  LI  by  discussing: 

•  How  to  Prepare  for  Analysis  -  Is  further  analysis  required?  Which  approach  will 
you  use?  Do  you  need  assistance?  What  additional  information  might  you 
need? 

•  Visualization  -  What  techniques  can  you  use  that  will  help  you  to  see  the 
patterns  in  your  data? 

•  Analysis  Techniques  -  What  methods  of  categorization  will  help  you  to  make 
sense  of  your  data?  What  do  the  statistics  tell  you?  How  can  you  compare 
different  potential  solutions? 

•  How  to  Write  Up  the  Lesson  Identified  -  How  will  you  document  the  results  of 
your  analysis  such  that  you  have  a  LI  ready  to  take  the  next  step  in  the  LL 
process? 

PREPARE  FOR  ANALYSIS 
How  mature  are  the  observations? 

The  first  step  in  the  analysis  step  is  to  examine  the  observation  to  answer  the  following 
questions: 

•  Does  the  observation  contain  any  causes  of  the  observed  issue  (i.e. 
explanations  of  why  the  issue  occurred)? 

•  Do  the  explanations  of  the  causes  (i.e.  why  it  happened)  seem  to  be  correct? 

•  Does  the  observation  contain  a  recommendation  (solution)  about  how  to 
address  the  cause  of  the  observed  issue? 

•  Does  the  proposed  recommendation  seem  to  address  the  causes  of  the 
observation?  Will  following  it  ensure  the  learning  becomes  ‘institutionalized’? 

If  the  answer  to  all  these  questions  is  yes,  then  the  observation  is  considered  to  be 
mature  and  no  further  analysis  of  the  issue  may  be  needed.  Once  the 
recommendation  is  written  into  an  RA  and  a  suitable  Action  Body  (AB)  is  identified,  the 
observation  can  be  considered  an  LI.  The  RA  and  AB  should  be  clearly  documented  in 
the  Recommendation  field  of  the  ODCR  template. 

However,  if  the  answer  to  any  of  these  questions  is  no,  then  the  observation  is 
considered  to  be  raw  and  further  analysis  of  the  observed  issue  is  required. 

Do  you  need  analysis  help? 

The  analysis  does  not  necessarily  need  to  be  carried  out  by  professional  analysts,  but 
does  require  staff  to  look  dispassionately  at  the  issue  to  identify  the  root  cause(s)  of  the 
problem.  In  some  cases,  observations  may  relate  to  issues  which  are  outside  the 
operational  control  of  the  originating  organization.  In  others,  the  originating 
organization  may  not  have  the  necessary  resources  or  subject  matter  expertise  to 
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address  the  issue.  It  is  important  to  recognize  when  such  observations  occur,  and 
request  analysis  through  the  chain  of  command. 

Within  NATO,  the  JALLC  performs  joint  analysis  of  operations,  exercises,  training 
events  and  experimentation.  To  obtain  assistance  from  the  JALLC  with  the  analysis  of 
a  specific  issue,  propose  an  analysis  requirement  to  either  SHAPE  or  HQ  SACT. 

JALLC  can  help  you  to  develop  an  analysis  requirement  relevant  to  your  specific  issue. 
If  the  work  is  determined  to  be  a  priority,  JALLC  will  be  tasked  with  the  analysis, 
through  either  the  annual  JALLC  programme  of  work  or  an  emergent  analysis 
requirement.  Additionally,  JALLC  analysts  are  available  to  provide  advice  and 
assistance  regarding  analysis  tools  and  techniques.  This  assistance  can  be 
coordinated  through  the  JALLC  Production  Branch,  by  email  at 
tlcqpx0010@iallc.nato.int. 

JALLC  holds  a  week-long  analyst  training  course  twice  a  year,  usually  in  April  and 
September.  These  courses  are  designed  to  train  military  analysts  assigned  to  the 
JALLC  on  the  LL  analysis  process.  However,  the  course  content  is  relevant  to  LLSOs 
and  there  are  a  limited  number  of  seats  available  to  external  participants.  The  point  of 
contact  is  JALLC  Lessons  Learned  Analysis  Branch,  by  email  at 
tlcqkx0010@iallc.nato.int. 

General  approach  to  analysis:  deductive  or  inductive  reasoning? 

There  are  two  broad  methods  of  reasoning  that  can  guide  your  analysis  approach; 
deductive  and  inductive  reasoning.  For  each  approach,  the  ultimate  goal  is  to 
determine  the  root  cause(s)  for  the  issues  described  in  observations. 

Deductive  reasoning  or  the  “top-down”  approach  (Figure  22)  begins  with  a  theory 
based  only  on  facts  shown  in  observations.  From  this  theory  one  or  more  hypotheses 
are  deduced  that  can  be  tested  by  further  observation.  If  the  hypotheses  are  proven 
correct,  it  suggests  the  original  theory  is  also  correct.  In  this  way,  the  deductive 
approach  leads  from  a  general  theory  to  more  specific  conclusions. 

Conversely,  inductive  reasoning  (Figure  3)  is  a  “bottom-up”  approach  where  many 
specific  observations  are  analysed  to  find  patterns  or  trends.  Patterns  are  then 
analysed  to  form  a  hypothesis,  and  the  hypothesis  eventually  facilitates  development  of 
a  theory.  Through  this  approach,  more  encompassing  theories  may  be  formed  from  a 
number  of  observations  that  would  otherwise  appear  unrelated. 


Observations 


Theory 


Tentative 

Hypothesis 


Pattern 


Figure  2:  Deductive  reasoning 


Figure  3:  Inductive  reasoning 
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Analysis  in  the  LL  process  often  follows  an  inductive  approach,  where  patterns  that 
suggest  a  hypothesis  emerge  from  observations.  The  hypothesis  is  then  tested  to 
arrive  at  a  theory  regarding  root  causes  for  issues  described  in  the  original  observation. 

Do  you  need  more  information? 

The  inductive  approach  may  require  more  observations  or  more  information  about  the 
observations  before  patterns  can  be  found.  For  this  reason,  you  may  be  required  to 
collect  additional  data  to  facilitate  your  analysis. 

One  way  of  collecting  additional  information  is  the  use  of  interviews.  The  Joint  Analysis 
Handbook  (Reference  G)  provides  guidance  on  interview  techniques.  There  are  three 
general  types  of  interviews  described: 
structured,  semi-structured  and 
unstructured.  The  use  of  structured 
interviews  is  recommended  when  gathering 
information  for  LL  analysis.  Structured 
interviews  use  standardized  questions  that 
are  identical  for  each  interview.  In  this  way 
they  are  similar  to  questionnaires  or  surveys,  except  the  interviewer  guides  the 
interviewee  through  each  of  the  questions  and  records  the  responses. 

Questionnaires  are  another  way  of  collecting  additional  data  that  are  usually  used 
when  you  wish  to  collect  the  same  information  from  a  large  number  of  respondents. 
When  additional  data  is  needed  about  an  observation  provided  by  an  individual, 
questionnaire-style  data  collection  is  usually  not  as  effective  as  a  structured  interview  - 
use  a  structured  interview  instead. 

Preparation  is  critical  for  success:  A  data  collection  plan  should  be  prepared  prior  to 
the  interview  or  questionnaire  and  questions  must  be  carefully  designed  to  obtain  the 
desired  data.  An  interview  process  is  further  described  in  Annex  D.  The  Joint  Analysis 
Handbook  (Reference  G)  and  Research  Methods  for  Business  Students  (Reference  H) 
are  good  resources  for  developing  interviews  and  questionnaires. 

VISUALIZATION 

Diagrams  provide  an  easy  way  to  visualize  information  and  explore  relationships  that 
would  otherwise  not  be  apparent.  The  Joint  Analysis  Handbook  describes  various  data 
visualization  models  that  can  be  used  to  facilitate  further  analysis.  Most  commonly 
used  in  LL  analysis  are: 

Cause  and  effect 

The  cause  and  effect  diagram  ( Fishbone  chart  or  Ishikawa  chart)  is  used  to: 

•  Focus  attention  on  one  specific  issue. 

•  Organize  and  display  graphically  the  various  theories  about  what  the  root 
causes  of  an  issue  may  be. 

•  Show  the  relationship  of  various  factors  influencing  an  issue. 

•  Reveal  important  relationships  between  possible  causes. 

•  Provide  additional  insight  into  process  behaviours. 

•  Focus  the  analysis  on  the  causes,  not  the  effects  or  symptoms. 

Flowcharts 

Flowcharts  are  used  to  represent  a  process  broken  down  into  less  complicated  sub¬ 
processes.  By  describing  only  a  limited  number  of  steps  or  activities  at  any  one  stage, 
the  overall  process  becomes  more  manageable  and  understandable.  Cross-functional 
flowcharts  (“swimlanes”)  are  used  to  illustrate  which  part  of  an  organization  performs 


Tip:  Interviews 

When  conducting  an  interview,  start  by 
trying  to  establish  a  good  relationship  with 
the  interviewee.  Explain  how  the  interview 
contributes  to  the  bigger  picture.  Close  the 
Interview  by  asking  “is  there  anything  else 
you  had  hoped  I  would  ask?”  or  similar. 
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particular  activities  or  functions,  and  are  useful  in  understanding  organizational 
relationships. 

Influence  diagrams 

Influence  diagrams,  or  systems  diagrams,  are  particularly  useful  in  identification  of 
logical  relationships  that  may  exist  within  the  observation  data  and  for  mapping  the 
logical  thought  process. 

ANALYSIS  TECHNIQUES 

The  Joint  Analysis  Handbook  provides  information  on  various  analysis  techniques. 

The  following  section  highlights  some  techniques  that  are  easily  applied  to  analysis  in 
support  of  an  LL  process.  Some  techniques  are  better-suited  for  finding  root  causes, 
some  are  better  for  recommending  RA’s,  and  some  can  be  used  for  both  purposes. 

The  techniques  used  can  be  adapted  to  meet  your  specific  needs. 

Five  Ws 

Simply  answer  What,  Where,  When,  Why,  Who,  How?  This  technique  is  easily 
understood  and  facilitates  information  gathering  and  investigation. 

Five  reasons  why  /  five  times  why 

This  technique  is  a  form  of  deductive  reasoning  that  is  often  used  to  support  root  cause 
analysis.  The  technique  can  be  used  on  an  individual  or  group  basis.  It  consists  of  the 
following  steps: 

Step  1:  Clearly  state  the  issue  identified  in  the  observation. 

Step  2:  Brainstorm  five  reasons  why  the  issue  occurred. 

Step  3:  For  each  of  the  five  reasons,  answer  the  question  “Why  has  this  happened?” 
Step  4:  Write  down  the  answer  to  these  question. 

Step  5:  For  the  answers  given  you  should  again  ask  why,  repeating  this  process  five 
times. 

Organization,  Process,  Technology  Categorization 

Organization,  Process,  Technology  (OPT)  is  a  technique  in  which  issues  are 
categorized  as  a  process  issue,  organizational  issue,  technology  issue;  or  some 
combination  of  these.  The  act  of  categorizing  the  issue  along  these  lines  will  often 
clarify  ideas  and  help  identify  root  causes. 

Doctrine,  Organization,  Training,  Materiel,  Leadership,  Personnel,  Facilities- 
Interoperability  (DOTMLPF-I)  Capability  Categorization 

DOTMLPF-I  categories  are  used  by  NATO  to  guide  capability  development. 
Categorizing  issues  using  DOTMLPF-I  is  another  approach  to  the  analysis  of  an 
observed  issue.  Using  DOTMLPF-I  categorization  is  particularly  useful  when 
developing  the  RA,  as  it  comprehensively  covers  considerations  for  developing  or 
refining  capabilities. 

Statistical  Analysis 

Statistical  analysis  is  a  field  of  mathematics 
related  to  drawing  conclusions  about  a  large 
population  based  on  limited  sample  data. 

You  should  have  some  familiarity  with 
statistical  methods  and  be  able  to  summarize 


Tip:  Statistical  Analysis 

The  Joint  Analysis  Handbook  (Reference  E) 
gives  information  about  statistical  analysis. 
Additionally,  Excel  provides  information  in 
its  Help  file  (press  FI  or  select  the  help 
menu  and  search  for  “statistical  analysis”). 
Also,  do  not  hesitate  to  ask  for  help  -OA 
staff  love  to  show  off  their  prowess  at 
statistics. 
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data  with  descriptive  statistics.  Specifically,  you  should  have  some  understanding  of: 

•  The  distinction  between  qualitative  variables  (either  categorical,  e.g.  military 
services,  or  ordinal,  e.g.  military  rank)  and  quantitative  variables  (numbers). 

•  Descriptive  statistics  including:  range,  mean,  mode,  median,  standard  deviation, 
etc.,  and  how  to  find  them  using  a  calculator  or  a  spreadsheet  application. 

•  Proportions,  ratios,  and  percentages. 

•  Data  representation  using  pie  charts,  bar  charts,  histograms  and  x-y  scatter 
plots  and  which  are  appropriate  for  your  data. 

Six  thinking  hats 

The  six  thinking  hats  method  is  used  to  explore  an  issue  using  the  six  main  modes  of 
thinking,  ultimately  giving  you  different  points  of  view.  A  technique  for  a  group,  it 
supposes  the  existence  of  six  hats  in  different  colours,  each  one  representing  a 
different  way  of  thinking.  When  you  figuratively  put  on  one  of  the  hats,  it  is  mandatory 
to  think  only  in  that  particular  way.  Each  person  selects  a  hat  and  ideas  are  discussed 
Hats  are  then  changed  until  everyone  in  the  group  has  worn  all  six  hats.  At  the  end  of 
the  session,  the  participants  record  their  ideas  and  apply  them  to  solving  the  problem. 

Plus/Minus/Interesting 

Plus/Minus/Interesting  (PMI)  is  a  variation  on  the  more  common  advantages  and 
disadvantages  technique,  by  introducing  a  third  category:  interesting.  The  interesting 
category  is  used  to  record  all  possible  outcomes  and  implications  of  adopting  a 
strategy,  whether  positive  (advantage),  negative  (disadvantage),  or  uncertain. 

Paired  comparison  analysis 

Paired  comparison  is  used  to  support  course-of-action  analysis  and  compares  each 
possible  pair  of  possible  solutions  (courses  of  action)  against  a  number  of  factors.  It  is 
a  good  way  of  weighing  the  relative  importance  of  the  different  courses  of  action 
against  one  or  more  factors  and  is  useful  when  priorities  are  not  clear. 


Tip:  Analysis  Resources 

Formal  analysis  is  a  huge  subject  area  and  the  methods  described  above  represent  just  a  small 
selection  of  techniques  that  are  useful  to  support  an  LL  process.  There  are  many,  many  others  that 
you  may  want  to  think  about.  Helpful  resources  include:  the  Joint  Analysis  Handbook  (Reference  G) 
which  you  can  download  from  the  JALLC  website  at  www.iallc.nato.int:  Research  Methods  for 
Business  Students  (Reference  H);  the  Mind  Tools  website  at  www.  mindtools.  com:  or  Wikipedia  at 
www. wikipedia.com.  Always  remember  that  a  bit  of  thinking  and  common  sense  will  always  go  a  long 
way  to  solving  issues. 


WRITE  UP  LESSONS  IDENTIFIED 

The  aim  of  the  analysis  step  is  to  provide  an  explanation  of  why  the  issue  described  by 
the  observation  occurred — i.e.  the  root  cause — and  provide  a  solution  to  fix  the  issue. 
After  the  analysis,  the  ODCR  template  can  be  completed  to  record  the  resultant  Lis. 
Additional  guidance  on  using  the  ODCR  template  can  be  found  in  Annex  B. 

Observation:  Supplementary  information  should  be  added  to  the  Observation  field  if 
necessary  to  facilitate  analysis. 

Discussion:  The  analysis  process  should  be  described  in  the  Discussion  field.  Be  as 
concise  as  possible,  but  provide  detail  such  that  the  issue  and  subsequent  analysis  can 
be  understood  by  future  users. 

Conclusion:  In  the  Conclusion  field,  explain  the  root  cause(s)  of  the  observed  issue. 
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Recommendation:  Finally,  in  the  Recommendation  field,  describe  the  RA  and  specify 
the  AB. 

SUMMARY 

•  In  order  to  transition  an  observation  into  an  LI,  analysis  must  be  conducted  to 
determine  the  root  cause(s)  and  to  seek  a  solution  (an  RA  that  an  AB  will  carry 
out). 

•  Preparing  for  analysis  involves  identifying  which  observations  need  analysis, 
whether  you  need  help  with  the  analysis,  what  analysis  method  you  will  use  and 
what  additional  information  you  might  need. 

•  Diagrams  provide  an  easy  way  to  visualize  information  and  explore  relationships 
that  would  otherwise  not  be  apparent. 

•  The  techniques  used  for  analysis  will  vary  and  depend  on  each  individual  LL 
process.  Several  resources  are  provided  for  further  information  and/or 
assistance. 

•  The  results  of  your  analysis  are  documented  Lis  that  are  ready  to  be  taken  into 
the  next  step  in  the  LL  process. 
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5 

Staffing  Lessons  Identified  to  Lessons  Learned 

Once  an  LI  has  been  generated  with  a  suitable  proposed  RA  and  associated  AB,  effort 
will  focus  on  making  it  into  a  Lesson  Learned  rather  than  a  lesson  admired.  Staffing 
Lis  to  Lessons  Learned  relies  on  everyone  involved  following  the  principles  of  the  LL 
process  as  described  in  Bi-SC  Directive  on  LL  (Reference  C): 

•  “Cooperation  is  critical  in  ensuring  that  those  bodies  involved  in  learning  a 
lesson  play  a  full  part  and  assist  in  achieving  progress; 

•  Coordination  demands  from  each  body  to  plan  and  deconflict  their  efforts 
effectively  to  ensure  a  timely  and  effective  management  of  their  work; 

•  Communication  will  facilitate  this  and  ensure  that  all  bodies  are  informed  on 
progress.  It  will  allow  others  visibility  of  nascent  lessons  to  begin  to  appraise 
them  before  formal  change  takes  place.  ” 

The  task  of  turning  an  LI  into  a  Lesson  Learned  can  be  thought  of  as  a  project  and,  as 
with  any  project,  successful  completion  will  require  good  project  management.  This 
chapter  introduces  project  management  considerations  for  staffing  an  LI  to  a  Lesson 
Learned  and  looks  at  the  steps  involved.  Specific  consideration  is  given  to  the 
following: 

•  Project  Management  Considerations 

•  Endorsement  and  Tasking 

•  Implementation  and  Monitoring 

•  Tools  to  Support  Staffing  an  LI  to  a  Lesson  Learned 

PROJECT  MANAGEMENT  CONSIDERATIONS 

Project  management  best  practices  can  be  applied  to  improve  the  effectiveness  of 
staffing  an  LI  to  a  Lesson  Learned.  NATO  has  adopted  the  PRINCE2  approach  as  its 
project  management  standard.  You  can 
tailor  the  PRINCE2  approach  to  meet  the 
project  management  needs  of  your 
organization  when  it  comes  to  staffing  Lis  to 
LLs. 

The  following  PRINCE2  principles  must  be  considered  in  the  context  of  an  LL  process: 

•  Organization:  Who  will  oversee  the  RA?  How  will  the  RA  be  managed?  What 
resources  are  required  to  carry  out  the  work  that  is  needed  to  complete  the  RA? 
Where  will  the  resources  come  from? 

•  Plans:  What  level  of  detail  is  realistically  needed?  What  are  the  significant 
milestones  that  can  be  monitored? 

•  Controls:  What  controls  need  to  be  in  place  to  be  able  to  tell  whether  the  RA  is 
going  to  need  more  time,  more  resources,  or  if  the  final  completed  RA  will  be  of 
a  lower  quality  than  originally  anticipated? 

•  Management  of  risk:  What  risks  may  prevent  the  RA  from  being  achieved 
successfully?  What  is  the  probability  and  severity  of  those  risks? 

•  Quality  Assurance:  How  will  you  ensure  the  project  achieves  the  necessary 
level  of  quality?  How  many  checks  and  balances  are  needed?  What 
requirements  should  the  AB  meet  to  prove  the  quality  of  the  solution? 


Tip:  PRINCE2 

Although  the  formal  application  of  PRINCE2 
is  probably  not  warranted  in  an  LL  process, 
knowledge  of  the  basic  principles  is 
valuable,  www.orince2.com 
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Other  considerations  that  have  proved  important  in  the  military  environment  are: 

•  Leadership  support:  Leadership  support  is  critical  in  the  endorsement  of  the  RA 
and  tasking  of  the  AB.  Without  command  direction  on  the  RA  and  AB,  the 
lesson  will  likely  stall  in  the  LL  process  as  the  organization  will  fail  to  complete 
the  action  necessary  to  ‘institutionalize’  the  learning. 

•  Clarity  of  roles  and  responsibilities:  Participants  must  understand  their  roles  and 
appreciate  how  they  fit  into  the  LL  process.  Care  should  be  taken  to  ensure  the 
process  is  easily  understood  and  adequately  explained. 

•  Prioritization  of  resources:  A  process  of  prioritization  of  Lis  will  help  to  ensure 
leaders  are  able  to  make  informed  decisions  regarding  how  many  resources  to 
allocate  to  turning  an  LI  into  a  Lesson  Learned. 

•  Method  of  communication:  Communication  of  information  must  be  simple, 
accessible,  and  timely.  Part  of  the  communication  process  may  involve  setup  of 
an  archive  for  LL  information  that  will  be  accessible  to  all  who  need  it. 

ENDORSEMENT  AND  TASKING 

The  endorsement  and  tasking  of  an  AB  to  complete  the  RA  is  the  first  step  in  turning  an 
LI  into  a  Lesson  Learned.  Specifically: 

•  Endorsement  of  a  lesson  means  it  has  been  approved  by  the  authorized 
decision-making  body.  This  implies  a  process  of  review  that  includes  checks  for 
completeness  and  accuracy  with  respect  to  root  cause(s)  and  consideration  of 
recommended  RA(s). 

•  Tasking  of  an  AB  means  an  AB  is  selected  and  will  be  responsible  for 
implementing  the  RA.  It  is  important  to  consider  at  this  point  who  has  the 
authority  to  task  an  AB.  The  level  of  authority  required  will  vary  depending  on 
the  authority  needed  to  execute  the  RA.  It  is  important  that  the  LI  is  elevated  to 
the  appropriate  level  to  ensure  that  the  AB  will  have  adequate  authority, 
jurisdiction,  etc. 

The  LLSO’s  role  in  supporting  endorsement  and  tasking  will  usually  be  to  prepare  Lis 
for  presentation  to  the  decision  makers  and  to  coordinate  and  administer  meetings 
where  endorsement  and  tasking  takes  place. 

IMPLEMENTATION  AND  MONITORING 

Once  endorsement  and  tasking  is  complete,  it  is  time  for  implementation  and 
monitoring  of  the  RA.  The  AB  tasked  with  the  RA  should  develop  a  RA  plan  for 
implementation.  To  assist  in  the  monitoring  of  the  RA  plan,  a  small  number  of 
significant  milestones  should  be  defined.  The  leadership  should  monitor  these  key 
milestones  to  measure  success  of  the  RA  plan  implementation. 

The  LLSO’s  role  in  supporting  the  implementation  and  monitoring  will  usually  focus  on 
monitoring.  You  will  usually  be  responsible  for  knowing  the  status  of  all  RAs  being 
implemented  and  keeping  leadership  informed  of  any  cost,  schedule,  or  management 
risks  to  implementation. 

VALIDATION 

Validation  in  the  context  of  the  NATO  Bi-SC  LL  process  is  the  act  of  ensuring  the 
completed  RA  has  correctly  addressed  the  original  issue  observed.  The  process  and 
level  of  effort  required  to  validate  will  be  determined  on  a  case-by-case  basis.  Factors 
to  consider  include: 

•  Impact  of  the  RA:  RAs  affecting  mission-critical  items  may  require  more  in  depth 
validation  before  being  deployed. 
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•  Extent  of  RA:  RAs  with  potential  wide-ranging  effects  may  require  more  in  depth 
validation. 

In  most  cases,  a  third  party  SME  (i.e.  independent  of  the  AB)  should  be  consulted  to 
evaluate  whether  the  RA  has  had  the  desired  effect.  Expert  validation  is  often 
sufficient  to  accept  the  RA  and  in  this  case  an  in-depth  validation  by  analysis  or 
experiment  is  not  required.  If  in-depth  validation  is  needed,  a  request  should  be  made 
to  external  agencies  to  support  analysis  or  experimentation  to  assess  whether  the 
lesson  has  been  learned. 


Tip:  Remedial  Action  Pitfalls 

The  RA  process  is  susceptible  to  many  risks  that  can  delay  or  halt  completion  of  the  RA. 
Common  pitfalls  include  inappropriate  or  circuitous  business  processes,  lack  of  quality  staffing, 
lack  of  adequate  resources,  and  lack  of  adequate  training  for  staff  involved  in  the  process. 
Leadership  engagement  greatly  mitigates  these  risks.  Where  leadership  engagement  is  good, 
LLSOs  will  have  a  direct  line  of  communication  into  the  command  group  and  ideally  work 
directly  for  their  organization’s  Chief-of-Staff . 


TOOLS  TO  SUPPORT  STAFFING  LI  TO  LL 

Throughout  the  LL  process,  a  structured  means  to  track  and  document  progress  of 
lessons  is  needed.  When  a  Lesson  Learned  is  achieved,  it  should  be  recorded  as  such 
in  all  supporting  information  and  documentation,  otherwise  it  can  seem  as  if  the 
organization  is  not  learning  anything. 

Organizations  create  or  adapt  various  tools  for  this  purpose,  including  locally- 
developed  spreadsheets  and  databases,  formal  command  correspondence,  and  other 
task  tracking  systems.  The  NATO  LLDb  is  designed  specifically  to  support  this 
function,  and  is  easily  adapted  to  meet  local  command  needs.  By  installing  a  copy  on 
your  local  network,  you  can  keep  track  of  internal  lessons  without  sharing  them  with  the 
rest  of  NATO  before  you  are  ready. 

Locally  developed  spreadsheets  and  databases 

The  advantage  of  using  a  spreadsheet  is  that  you  can  see  all  of  the  data  in  one  go,  this 
means  that  even  if  people  are  not  very  computer  literate,  they  will  be  able  to  use  the 
spreadsheet  properly  without  too  much  difficulty.  Spreadsheets  can  also  filter  data 
very  simply  using  drop  down  menus  so  you  can  view  only  Lis  with  a  certain  status  at 
any  time.  One  of  the  big  disadvantages  of  using  a  spreadsheet  with  lessons  entered 
as  individual  rows  is  that  as  the  information  to  support  the  RA  process  is  added  to  the 
basic  ODOR  template — e.g.  ABs,  milestones,  etc — more  and  more  columns  get  used 
and  the  sheet  becomes  increasingly  unmanageable. 

The  advantage  of  using  databases  is  that  they  generally  have  excellent  form 
generation  and  reporting  capabilities,  and  are  relatively  easy  to  customize  and 
program.  A  common  pitfall  to  using  a  database  is  that  as  soon  as  the  person  that 
developed  it  leaves  the  organization,  it  falls  into  disuse  because  there  is  inadequate 
supporting  documentation  and  the  overall  design  (e.g.  the  way  the  tables  are  set  up)  is 
not  intuitive  to  a  third  person. 

Letters,  Memos  and  Tasker-Tracker  systems 

The  advantage  of  using  letters,  memos  or  tasker-tracker  systems  is  that  they  are  pre¬ 
existing  means  of  communication  within  an  organization.  This  means  the  LL  process  is 
leveraging  existing  infrastructure  rather  than  requiring  development  of  additional  tools. 
Using  these  tools  helps  to  integrate  LL  into  part  of  everyday  business  rather  than  as  a 
special  tasking.  The  disadvantage  of  using  these  tools  is  that  they  may  not  provide 
needed  functionalities  -  such  as  ability  to  attach  lesson  metadata.  You  should  ensure 
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that  if  you  use  these  tools,  it  is  mandatory  for  the  letter,  memo  or  tasker-tracker  entry  to 
specifically  reference  the  LI  to  which  it  is  related. 

The  NATO  Lessons  Learned  Database 

The  NATO  LLDb  is  a  powerful  tool  available  to  all  NATO  organizations,  NATO  nations 
and  partner  nations.  It  runs  on  a  server  allowing  reliable  access  for  multiple  users,  but 
requires  Microsoft  SQL  Server  to  run  and  thus 
requires  significant  technical  support  and 
resources  for  its  initial  set  up. 

The  NATO  LLDb  supports  monitoring  of  all 
the  steps  of  an  entire  LL  process.  The 
database  acts  as  a  staffing  tool:  facilitating 
the  cooperation  of  all  participants, 
coordination  of  their  actions,  and 
communication  of  lessons  and  RA.  More  information  about  the  NATO  LLDb  can  be 
found  at  www.iallc.nato.int. 

SUMMARY 

•  Leadership  engagement  is  key  to  the  process  of  staffing  an  LI  to  a  Lesson 
Learned.  The  appropriate  level  of  leadership  needs  to  be  involved  to  endorse 
the  lesson,  task  the  AB  and  finally  validate  the  lesson.  Additionally,  leadership’s 
prioritization  will  initially  determine  whether  the  lesson  will  be  staffed.  Only 
adequate  resourcing  of  the  LL  process  will  ensure  success. 

•  Implementation  and  monitoring  of  the  RA  is  a  project  which  needs  to  be 
planned,  managed,  and  resourced  in  order  to  be  successful. 

•  It  is  important  to  have  a  tool  that  is  easily  understood  and  familiar  to  all  users  to 
staff  lessons  within  the  LL  process.  Spreadsheets,  databases  and  existing 
tasker-tracker  tools  provide  a  means  to  manage  the  LL  process. 

•  The  NATO  LLDb  is  a  managed  LLDb  which  supports  all  the  steps  of  an  LL 
process.  It  is  a  good  tool  to  support  staffing  of  an  LL  process. 


Tip:  NATO  LLDb 

The  NATO  LLDb  Quick  Start  Guide  and 
other  resources,  such  as  the  Microsoft 
PowerPoint  briefs  on  Using  the  LLDb  and 
Staffing  in  the  LLDb,  are  available  via  the 
LLDb,  which  can  be  found  on  the  NS  WAN 
at  nww.iallc.nato.int  and  on  the  internet  at 
www.  iallc.  nato.  int. 
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6 

Lesson  Learned  Information  Sharing 

The  value  of  an  LL  process  is  only  realized  when  the  information  generated  by  the 
process  is  available  to  the  people  who  need  it,  when  they  need  it.  LL  information 
sharing  generates  organizational  knowledge  and  leads  to  an  enduring  improvement  in 
organizational  performance.  But  not  everyone  is  motivated  to  share.  Commonly- 
expressed  reasons  for  not  sharing  include: 

•  Sharing  negative  experiences  creates 
embarrassment  and/or  blame 

•  It  is  not  worth  sharing  until  we  have  a 
solution 

•  Sharing  information  is  a  risk: 
information  obtained  by  the  enemy 
could  be  used  to  exploit  our 
weaknesses. 

•  Lessons  can  only  be  learned  by  doing: 
documenting  experiences  is  a  waste  of 
time. 

•  The  lessons  are  classified  and  we 
cannot  change  that  to  share  them. 

However,  there  are  great  benefits  to 
overcoming  these  concerns  and  sharing.  Sharing  knowledge  yields  better  results  in 
both  business  and  military  environments.  In  military  terms,  this  means  both  saving 
lives  and  succeeding  in  the  mission.  With  proper  information  management,  all  of  the 
above  concerns  are  mitigated  and  become  far  outweighed  by  the  benefits  of  sharing 
the  information. 

The  LLSO  is  responsible  for  the  organization’s  LL  information  sharing  and  will  need  to 
understand: 

•  Who  to  share  LL  information  with? 

•  When  to  share  information  and  at  what  stage  in  the  process  do  you  share? 

What  LL  information  to  share? 

•  How  to  share  LL  information?  Communities  of  interest,  forums  (working  groups, 
conferences,  etc.),  training,  tools  (databases,  wikis,  etc.),  publications 
(newsletters,  reports,  etc.) 

This  chapter  answers  these  questions  and  can  guide  you  in  making  your  organization’s 
LL  information  sharing  as  effective  as  possible. 

WHO  TO  SHARE  LL  INFORMATION  WITH? 

Sharing  is  a  verb  that  means  to  “tell  someone  about”12.  In  sharing  a  lesson  learned,  it 
is  not  enough  to  simply  publicize  it.  Some  consideration  must  be  given  to  who  will 
benefit  from  the  lesson,  and  this  group  is  referred  to  as  your  target  audience.  Care 
should  be  taken  when  sharing  lessons  to  ensure  relevance  to  the  target  audience  and 
therefore  promote  effective  learning.  The  way  you  present  LL  information  to  a  General 


Figure  4:  LL  Tombstone 
(Reference  B) 


Tip:  The  Power  of  Sharing  Knowledge 

If  you  want  to  be  convinced  about  the 
power  of  sharing  knowledge,  visit 
www.knoco.com  and  read  about  the  Bird 
Island  exercise! 


12  Definition  2  from  Concise  Oxford  English  Dictionary  11th  Edition. 
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who  needs  the  information  to  make  a  decision 
that  will  affect  the  entire  organization  will  need  to 
be  different  to  the  way  you  present  LL 
information  to  a  Private  who  needs  the 
information  to  improve  his  own  daily  working 
practices. 

In  a  military  organization,  lessons  will  be  a 
valuable  input  to  operations  and  exercise 
planning  processes  and  training.  The  use  of 
lessons  in  these  areas  should  be  routine.  These 
audiences  will  have  specific  requirements: 

•  Exercise  Planners:  Exercise  planners  should  review  previous  lessons  during  the 
exercise  planning  process.  They  form  the  groundwork  for  the  exercise  planning 
process  described  in  the  Bi-SC  Exercise  Directive  (Reference  E).  Lessons  most 
relevant  to  this  audience  are  likely  to  come  from  previous  exercises  and  final 
exercise  reports.  Operational  lessons  should  be  incorporated  if  possible. 

•  Operational  Planners:  Operational  planners  should  review  and  apply  lessons  in 
the  preparation,  planning  and  conduct  of  combat  operations.  Lessons  most 
relevant  to  the  audience  are  likely  to  come  from  previous  operations. 

•  Training:  Trainers  will  need  access  to  lessons  from  both  exercises  and 
operations  to  incorporate  immediately 
into  pre-deployment  training.  It  is 
particularly  important  that  lessons  are 
communicated  in  a  timely  fashion  to 
follow-on  forces  during  pre-deployment 
training. 

WHEN  TO  SHARE? 

LL  information  sharing  can  occur  at  any  time  during  the  LL  process,  not  just  as  the  last 
step  or  as  part  of  RA  implementation.  Additionally,  it  may  involve  sharing  information 
that  is  not  formally  part  of  an  LL  process  such  as  after  action  reviews,  periodic  mission 
reports,  first  impression  reports,  final  exercise  reports,  trip  reports,  hot  wash  up  output, 
meeting  minutes,  etc.  In  accordance  with  Bi-SC  Directive  80-6  (Reference  C),  the 
decision  to  share  an  observation  outside  an  organization  lies  with  the  commander. 

Security:  Need  to  Know  versus  Responsibility  to  Share 

According  to  NATO  Information  Management  Policy13,  information  should  be  managed 
with  emphasis  on  sharing,  but  with  due  consideration  of  security: 

“Information  shall  be  managed  with  an  emphasis  on  ‘responsibility-to-share’ 
balanced  by  the  security  principle  of  ’need-to-know’,  and  managed  to  facilitate 
access,  optimise  information  sharing  and  re-use,  and  reduce  duplication,  all  in 
accordance  with  security,  legal  and  privacy  obligations.  ” 

LL  information  is  no  different.  Sharing  lessons  relating  to  capabilities  (or  supposed 
vulnerabilities)  of  combat  forces  may  result  in  inadvertent  disclosure  of  classified 
information  to  someone  who  does  not  have  a  “need-to-know”  if  this  is  not  managed 
properly.  The  risk  of  unauthorized  disclosure  must  be  balanced  against  the  benefit  that 
could  be  achieved  through  well-managed  sharing.  In  multinational  units  or  where 
nations  work  together  with  adjacent  areas  of  responsibility  (e.g.  ISAF  operations), 
“responsibility-to-share”  is  very  important.  Knowledge  represented  by  lessons  must  be 

13  North  Atlantic  Council,  NATO  Information  Management  Policy,  11  December  2007,  C- 
M(2007)01 18,  NATO  Unclassified. 


Quote:  General  Mattis,  SACT 

“...There  is  no  reason  to  send  troops  into 
the  fight  and  get  them  killed  when  a  lesson 
learned  the  month  before  could  be  sent  to  a 
commander  who  could  have  used  it  for 
training..."  16  January  2009,  All  Hands  Call 


Tip:  Audience  is  everything 

The  target  audience  will  be  your  primary 
consideration  for  information  sharing.  For 
example,  a  lesson  learned  about  an 
internal  administrative  practice  unique  to  a 
small  command  may  be  of  little 
consequence  to  an  audience  of  ISAF 
combatant  troops.  By  the  same  token,  a 
tactical  observation  likely  to  recur  across 
NATO  that  could  prevent  combat  deaths 
would  be  of  limited  use  to  an  audience  of 
administrators. 
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shared  as  effectively  as  possible  among  nations  to  improve  the  effectiveness  and 
safety  of  all  units  involved. 

Good  and  Bad  Practice 

When  a  good  practice  is  noted,  there  is  a  natural  desire  to  tell  everybody  about  it 
immediately.  This  is  understandable,  but  should  be  done  with  caution  until  the  practice 
has  been  analyzed  properly  to  determine  the  conditions  and  circumstances  where  it  is 
valid,  and  how  it  can  be  ‘institutionalized’  smoothly.  The  danger  with  sharing  good 
practices  too  early  is  that  people  may  assume  this  is  enough  to  reach  a  Lesson 
Learned  and  then  take  no  further  action  to  ‘institutionalize’  them.  Simply  sharing  a 
good  practice  will  not  ensure  the  lesson  is  learned. 

Conversely,  there  may  be  a  natural  desire  to  hide  or  minimize  ineffective  or  detrimental 
practices,  or  to  blame  negative  outcomes  on  human  error  rather  than  ineffective  tactics, 
techniques,  or  procedures.  A  significant  part  of  your  role  as  LLSO  will  be  to  encourage 
sharing  of  mistakes  and  to  make  the  distinction  between  simple  human  error  and  more 
systemic  problems.  When  an  ineffective  or  detrimental  practice  is  concealed  or 
minimized,  it  denies  others  the  opportunity  to  learn  from  it,  and  it  restricts  the 
opportunity  to  use  knowledge  or  insights  gained  through  experience  to  improve. 

Quality  of  LL  Information 

Different  types  of  LL  Information  have  different  levels  of  quality.  Quality  is  dependant 
on  maturity  of  the  information  with  respect  to  the  LL  process,  in  other  words  the  amount 
of  analysis  and  scrutiny  it  has  undergone.  The  level  of  quality  will  affect  your  inclination 
to  share  the  information  as  well  as  its  utility  to  your  target  audience.  Examples  of 
information  at  differing  levels  of  quality  include: 

•  Low  quality  LL  information:  Raw  observations,  good  practices,  hot  wash  up 
output. 

•  Medium  quality  LL  information:  Newsletters,  mature  observations,  Lis,  first 
impression  reports. 

•  High  quality  LL  information:  Lessons  Learned,  LL  analysis  reports,  handbooks, 
final  exercise  report. 

You  will  probably  be  less  inclined  to  share  low  quality  LL  information,  because  you  may 
not  be  as  confident  in  the  veracity  of  the  information.  Low  quality  LL  information  may 
be  incomplete  or  factually  incorrect,  but  that  does  not  necessarily  mean  it  has  no  value. 
The  important  thing  to  remember  is  to  disclose  the  reliability  and  maturity  of  the 
information  to  others  to  ensure  they  can  use  it  appropriately.  Low  quality  LL 
information  may  be  useful  to  others  as  a  starting  point  for  further  planning, 
experimentation,  testing,  etc.  However,  it  should  not  be  acted  on  without  appropriate 
scrutiny  or  due  diligence,  as  doing  so  might  be  wasteful  or  dangerous.  Provided  the 
people  you  are  sharing  with  are  aware  of  the  quality  of  your  information,  they  will  be 
able  to  make  informed  decisions  about  how  to  best  use  that  information. 

HOW  TO  SHARE  LL  INFORMATION? 

Consideration  needs  to  be  given  to  the  pushing  and  pulling  of  information.  Pushing 
information  means  that  new  information  is  actively  sent  out  to  consumers  or 
subscribers  as  it  becomes  available,  while  pulling  information  implies  that  consumers 
have  to  regularly  check  to  see  if  new  information  has  become  available.  An  example  of 
pushing  information  is  the  distribution  of  newsletters  and  the  sending  of  e-mails  to 
subscribers  when  something  happens  like  the  posting  of  new  information  on  a  portal. 

An  example  of  pulling  information  is  publishing  it  to  a  database  where  people  are 
expected  to  go  to  find  the  information  without  being  alerted  that  new  or  updated 
information  has  been  published. 
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Most  organizations  will  choose  to  use  a  combined  push  and  pull  approach  whereby 
there  are  procedures  to  ensure  that  high-impact,  high-priority  and  urgent  information  is 
pushed  to  the  appropriate  people  quickly  and  lower-priority  issues  are  automatically 
stored  somewhere  until  the  user  chooses  to  check  for  updates.  An  example  of  the  use 
of  combined  push/pull  would  be  a  change  to  force  protection  alert  level  that  is  sent  via 
radio  to  all  force  protection  personnel  in  the  area  (push)  but  at  the  same  time  updated 
on  a  website  that  people  visit  to  check  for  updates  (pull). 

Communities  of  Interest 

Communities  of  interest  engage  in  socialized  learning  and  consist  of  people  who  have 
common  goals  interacting  with  each  other.  In  the  military  context,  opportunities  to 
interact  include  conferences,  working  level  meetings,  working  groups  and,  of  course 
direct  communication. 

There  are  a  number  of  benefits  to  participating  in  a  community  of  interest,  including: 

•  Solving  problems 

•  Developing  new  capabilities 

•  Leveraging  best  practices 

•  Standardizing  practices 

•  ‘Institutionalizing’  best  practices 

•  Time  saving 

•  Increasing  skill  sets 

•  Avoiding  mistakes 

•  Creating  new  knowledge 

There  are,  however,  a  number  of  obstacles  to  participating  in  a  community  of  interest, 
including  the  releasability  and  classification  of  information,  technological  problems  such 
as  connectivity,  and  ‘institutional  mindset  against  information  sharing.  Making  personal 
connections  (i.e.  networking)  significantly  enhances  your  ability  to  share  information. 
Personally  knowing  and  trusting  the  individual  from  whom  you  are  asking  for 
information,  or  to  whom  you  are  sending  information,  helps  information  sharing.  Much 
of  sharing  lessons,  particularly  across  national  boundaries,  is  via  informal  social 
networks  (the  community  of  interest)  rather  than  through  chain  of  command  and 
established  liaison  channels. 

Many  communities  of  interest  are  supported  by  forums  that  allow  them  to  share  their 
experiences.  Examples  of  forums  include  training  events,  conferences,  working  groups 
etc,  or  virtual  forums  online,  including  discussion  groups  and  blogs.  Conferences  not 
only  provide  information  but  also  offer  the  opportunity  to  network.  Working  groups  are 
generally  focussed  on  specific  areas,  subjects  or  stakeholder  groups,  for  instance  the 
NATO  International  Security  Assistance  Force  (ISAF)  Lessons  Learned  Support 
Working  Group,  or  the  NATO  Bi-SC  Medical  LL  Working  Group.  Participating  in  forums 
allows  for  reinforcing  personal  contacts  to  encourage  the  sharing  of  information. 

Request  for  Information  Service 

Some  organizations  offer  a  request  for  information  service,  where  individuals  requiring 
information  on  a  particular  topic  can  make  a  request  and  the  LL  branch  will  search  the 
LL  information  they  have  in  order  to  respond  to  the  request.  The  ability  to  respond  to 
the  request  will  often  depend  on  finding  the  right  POC  within  the  organization.  For  this 
reason  you  should  maintain  a  database  of  SME  POCs  within  your  organization.  The 
JALLC  maintains  a  central  database  of  LL  POCs  across  the  NATO  Command  and 
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Force  Structures  who  may  also  be  able  to  assist  you  in  responding  to  a  request  for  LL 
information.  JALLC  also  holds  LL  POC  details  in  some  NATO,  Partner  and  Troop 
Contributing  Nations. 

Training 

Training  events  at  the  beginning  of  rotations  into  theatre,  at  the  beginning  of  exercises, 
or  as  part  of  in-processing  into  a  new  billet  are  good  opportunities  to  engage  staff  on 
the  benefits,  opportunities  and  requirements  of  an  LL  process  as  well  as  to  inform  staff 
about  the  latest  lessons  from  the  field.  In  Kosovo  Force  (KFOR),  incoming  personnel 
attend  several  days  of  briefings  and  orientation  activities  which  introduce  them  to  their 
tour  of  duty.  This  is  an  ideal  opportunity  to  introduce  new  staff  to  lessons  from  previous 
rotations. 

Information  Technology 

There  are  several  information  technology  tools  that  support  the  sharing  of  lessons  and 
information,  including: 

•  Portals 

•  Databases,  for  example  the  NATO  LLDb 

•  Knowledge  repositories  such  as  wikis 

•  Blogs 

A  technology  solution  can  provide  access  to  many  different  types  of  LL  information 
easily.  However,  it  will  only  be  as  good  as  the  information  it  contains.  It  is  critical  that 
procedures  exist  to  ensure  that  it  is  populated  with  up-to-date,  relevant  information  that 
is  useful  to  staff  in  their  work. 

Publications 

In  addition  to  routine  reports  (e.g.  after  action  reviews,  periodic  mission  reports,  final 
exercise  reports,  etc),  there  are  several  ways  to  ensure  LL  information  reaches  those 
within  and  external  to  your  organization.  You  may  compile  information  into  regular: 

•  Newsletters 

•  Reports 

•  Booklets/Handbooks/Leaflets/Posters 

•  Email  Blasts 

•  Blogs/Bulletin  boards 


31 


SUMMARY 

Who  to  Share  With 

•  Carefully  select  the  target  audience  to  ensure  relevance  and  so  promote 
effective  learning. 

•  Lessons  are  a  valuable  input  to  operations  and  exercise  planning  processes  and 
therefore  operational  and  exercise  planners  are  one  target  audience  for  an 
organization’s  lessons. 

•  Lessons  are  a  valuable  input  to  training  and  so  trainers  are  another  target 
audience  for  an  organization’s  lessons. 

When  and  What  to  Share 

•  Sharing  knowledge  promotes  better  results  and  proper  information  management 
should  help  to  overcome  concerns  regarding  sharing. 

•  The  decision  to  share  an  observation  outside  an  organization  lies  with  the 
commander. 

•  All  LL  Information  can  be  shared  at  any  time,  as  long  as  it  is  clear  what  level  of 
quality  it  has. 

•  An  emphasis  on  “responsibility-to-share”  should  be  balanced  with  the  security 
principle  of  “need-to-know”. 

How  to  Share 

•  Consideration  needs  to  be  given  to  the  pushing  and  pulling  of  information. 
Pushing  actively  sends  out  new  information  to  individuals  as  it  becomes 
available.  Pulling  requires  the  individual  to  regularly  check  to  see  if  new 
information  is  available. 

•  Communities  of  interest  engage  in  socialized  learning  and  consist  of  people  who 
have  common  goals  interacting  with  each  other.  Information  may  be  shared 
within  the  community  via  forums,  working  groups  and  direct  communications. 

•  A  key  factor  in  sharing  information  is  making  the  effort  to  contribute  and 
reinforcing  personal  contacts.  Informal  sharing  via  social  networks  can 
complement  sharing  through  formal  military  channels. 

•  Training  events  are  good  opportunities  to  share  recent  lessons  from  the  field. 

•  Tools  that  support  the  sharing  of  lessons  and  information  include  databases,  for 
example  the  NATO  LLDb,  and  knowledge  repositories  such  as  wikis  and  blogs. 


32 


7 

Summary 

Congratulations  for  taking  the  time  to  read  this  handbook  which  has  walked  you 
through  the  essential  elements  of  a  successful  LL  process  and  highlighted  the 
importance  of  sharing  LL  information.  We  hope  that  you  have  found  this  handbook  to 
be  a  useful  and  practical  introduction  to  LL  that  has  improved  your  ability  to  effectively 
and  efficiently  support  LL  in  your  organisation. 

We  wish  to  keep  this  handbook  up-to-date  with  the  latest  policy,  procedures,  best 
practice  and  innovation  in  the  area  of  LL,  so  that  it  remains  a  genuinely  useful  resource 
for  everyone  working  in  LL.  If  you  notice  any  problems  with  the  handbook,  or  have  any 
best  practices  or  suggestions  about  how  to  improve  it,  please  email  them  to 
llh@iallc.nato.int. 

USEFUL  LESSONS  LEARNED  POINTS  OF  CONTACT 

•  To  contact  JALLC  for  LL  support,  email  iattpoc@iallc.nato.int 

FURTHER  LESSONS  LEARNED  INFORMATION 

For  further  information  on  NATO  LL  and  to  find  the  NATO  LLDb,  visit  the  NATO  LL 
Portal  at: 

•  Via  www.jallc.nato.int  on  the  internet 

•  Via  the  NATO  NS  WAN 

To  learn  more  about  LL  and  learning  organizations  in  general  refer  to: 

•  The  LL  Handbook  (Reference  B) 

•  The  Fifth  Discipline14 

To  brush  up  on  your  analysis  techniques  in  support  of  LL  see: 

•  Joint  Analysis  Handbook  (Reference  G) 

For  more  information  on  project  management  best  practice  for  staffing  LI  to  Lessons 
Learned: 

•  http://www.prince2.com 

To  learn  more  about  information  management  and  document  classification  and 
releasability  in  NATO: 

•  NATO  Security  Policy  documents15 


14  Senge,  P.,  The  Fifth  Discipline:  The  Art  and  Practice  of  the  Learning  Organization,  1st  Edition, 
Doubleday  Business,  1994,  ISBN:9780385260954. 

15  hww.hq.nato.int/NOS/en/library/index.asp  on  NS  WAN. 
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Annex  A 

_ LL  Glossary 

The  following  definitions  are  derived  from  the  fundamental  NATO  LL  documents 
(References  A,  C  and  D)  or  as  used  for  the  purpose  of  this  handbook: 


Action  Body  (AB) 
Analysis 

Best  Practice  (BP) 

Endorsement  and 
Tasking 

Gathering 

Observations 

Implementation  and 
Monitoring 

Lesson  Identified  (LI) 
Lesson  Learned 
Lessons  Learned  (LL) 

LL  Information 


Mature  Observation 

Metadata 

Observation 
Raw  Observation 

Remedial  Action  (RA) 

Validation 

Wiki 


An  organization  tasked  to  implement  or  facilitate  the 
implementation  of  an  RA. 

The  study  of  a  whole  by  thoroughly  examining  its  parts  and 
their  interactions. 

An  activity  or  a  series  of  activities  proven  effective  through 
analysis  that  can  be  replicated  by  others  in  a  similar  situation. 
The  step  in  a  LL  process  of  endorsing  the  LI  at  an  appropriate 
level  and  identifying  and  tasking  an  AB  to  develop  an  action 
plan  for  the  RA  and  take  responsibility  for  implementation. 

The  step  in  a  LL  process  of  gathering  issues  identified  for 
improvement  that  will  be  managed  through  the  LL  process, 
(proposed  definition) 

The  step  in  a  LL  process  of  the  AB  implementing  the  RA  and 
the  tasking  authority  monitoring  the  implementation 
An  observation  for  which  an  RA  has  been  developed  and  an 
AB  to  carry  out  the  remedial  action  has  been  proposed. 

(n.)  Results  from  the  implementation  of  a  RA  that  produced  an 
improved  performance  or  increased  capability. 

(adj.)  Of  or  relating  to  the  processes,  products  and  people  that 
ultimately  produce  a  lesson  learned,  (proposed  definition) 

Any  information  that  is  generated  as  part  of  an  LL  process  as 
well  as  information  generated  after  activities  that  is  not 
formally  part  of  an  LL  process  such  as  after  action  reviews, 
periodic  mission  reports,  first  impression  reports,  final  exercise 
reports,  trip  reports,  hot  wash  up  output,  meeting  minutes,  etc. 
(proposed  definition) 

An  observation  for  which  there  is  already  sufficient  data  and/or 
understanding  to  identify  the  root  causes  and  thus  requires  no 
further  analysis. 

A  set  of  data  that  describes  and  gives  information  about  other 
data.  (Concise  Oxford  English  Dictionary,  11th  Edition) 

An  issue  identified  for  improvement. 

An  observation  which  requires  further  study  or  analysis  to  fully 
understand  the  root  causes. 

A  possible  action  that  serves  to  rectify  a  fault  or  improve 
conditions. 

Determines  if  the  issue  that  was  originally  observed  has  been 
successfully  rectified  by  the  RA  carried  out  by  the  AB. 

A  website  that  allows  the  easy  creation  and  editing  of  any 
number  of  interlinked  web  pages  via  a  web  browser.  Wikis 
are  typically  powered  by  wiki  software  and  are  often  used  to 
create  collaborative  wiki  websites,  to  power  community 
websites,  for  personal  note  taking,  in  corporate  intranets,  and 
in  knowledge  management  systems.  (Wikipedia) 
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The  following  abbreviations  are  used  in  this  handbook: 


AB 

Action  Body 

ACO 

Allied  Command  Operations 

ACT 

Allied  Command  Transformation 

Bi-SC 

of  the  two  Strategic  Commands  (ACO  and  ACT) 

BP 

Best  Practice 

CALL 

Center  for  Army  Lessons  Learned  (US) 

CNAD 

Conference  of  National  Armaments  Directors 

COE 

Centre  of  Excellence 

DOTMLPF-I 

Doctrine,  Organization,  Training,  Materiel,  Leadership, 
Personnel,  Facilities  -  Interoperability 

IED 

Improvised  Explosive  Device 

IMS 

International  Military  Staff 

IS 

International  Staff 

ISAF 

International  Assistance  Force 

JALLC 

Joint  Analysis  and  Lessons  Learned  Centre 

JATT 

JALLC  Advisory  Training  Team 

JFC 

Joint  Force  Command 

KFOR 

Kosovo  Force 

LI 

Lesson  Identified 

LL 

Lessons  Learned  (adj.) 

LLDb 

Lessons  Learned  Database 

LLSO 

Lessons  Learned  Staff  Officer 

LLWG 

Lessons  Learned  Working  Group 

MC 

Military  Committee 

MCM 

Mine  Countermeasures 

OCP 

Observation  Collection  Program 

ODCR 

Observation,  Discussion,  Conclusion,  Recommendation 

OPT 

Organization,  Process,  Technology 

POC 

Point  of  Contact 

PPE 

Personal  Protective  Equipment 

PRINCE2 

Projects  In  Controlled  Environments  2 

RA 

Remedial  Action 

RSM 

Regimental  Sergeant  Major 

SACEUR 

Supreme  Allied  Commander  Europe 

SACT 

Supreme  Allied  Commander  Transformation 

SME 

Subject  Matter  Expert 

SOP 

Standing/Standard  Operating  Procedure 

SWEDINT 

Swedish  Armed  Forces  International  Centre 
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Annex  B 
Lesson  Template 


Title 


The  title  should  be  brief  but  specific.  It  should  give  a  reasonable  indication  as  to 
content  of  the  observation. 


Observation 

A  short  factual  statement  to  describe  what  happened 
and  how  that  differed  from  expectations.  This 
statement  can  be  positive  (i.e.  something  that  was 
observed  to  work  better  than  expected  or  a  work  around)  or  negative  (i.e.  something 
happened  that  should  not  have  or  something  did  not  happen  that  should  have).  Details 
should  be  presented  in  the  discussion  paragraph.  Observations  should  be  restricted  to 
single  issues.  Multiple  issues  should  be  divided  into  separate  observations  and  cross- 
referenced  to  each  other  in  the  discussion  section. 


Tip 

When  capturing  observations,  only 
the  observation  field  of  the 
template  is  mandatory 


WATCH  OUT!  Common  errors  include  listing  details  better  suited  for  the  discussion, 
conclusion,  or  recommendation  sections  of  the  template,  e.g.  “Staff  officers  should 
work  harder”,  or  including  too  little  information,  e.g.  “Lesson  345  was  not  learned  at  all”. 


Discussion 

The  discussion  explains  how  and  why  the  observed  issue  differed  from  expectations. 
Reasons  for  success  or  failure  and  the  circumstances  surrounding  the  issue  are 
discussed.  The  discussion  amplifies  the  observation  statement  and  answers  the,  “who, 
what,  where,  when,  why  and  how,”  questions  about  the  observation.  It  should  explore 
all  the  apparent  contributory  factors,  i.e.  the  analysis  of  the  observed  issue.  It  can 
include  the  history  of  the  event,  the  context  and  the  environment,  and  any  actions 
taken  to  work  around  a  problem  should  be  explained  in  detail.  If  a  problem  could  not  be 
solved  explain  why. 

WATCH  OUT!  Resist  the  temptation  to  repeat  the  observation.  Be  as  concise  as 
possible,  but  be  sure  to  include  all  data/information  you  expect  to  be  necessary  for 
further  analysis. 


Conclusion 

The  conclusion  is  a  summary  statement  of  the  lesson  that  has  been  learned  from  the 
experience  and  the  investigation  into  the  root  cause(s)  of  the  issues  described  in  the 
observation  and  discussion.  It  is  derived  in  a  logical  manner  from  the  information 
contained  in  the  observation  and  discussion. 

WATCH  OUT!  Avoid  too  much  detail,  and  make  sure  that  the  conclusion  contains  no 
new  information.  A  common  error  is  to  make  recommendations  instead  of  sticking 
purely  to  conclusions  about  root  cause(s).  Ensure  that  the  conclusion  follows  logically 
from  the  observation  and  the  discussion:  a  good  idea  is  to  get  someone  else  to  read  it 
and  make  sure  they  agree  with  your  logic.  Try  starting  off  the  conclusion  with  the 
phrase,  “Therefore,  we  have  learned  that...” 


Recommendation 

The  recommendation  should  outline  the  suggested  RA  by  providing  explicit  advice  on 
what  must  be  done  to  repeat  the  success  or  to  avoid  and/or  solve  the  problem.  Identify 
exactly  what  needs  to  be  changed — new  or  modified  publications,  procedures, 
procurement  of  new  equipment,  change  of  the  force  structure,  revision  of  command 
relationships,  improved  training,  etc. — and  how  this  should  be  done.  The 
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recommendation  should  also  propose  a  suitable  AB.  The  recommendation  should 
follow  logically  from  the  conclusion  so  that  if  someone  were  to  follow  the 
recommendation,  they  would  reap  the  benefit  of  the  learning  for  themselves  and  their 
organization. 

WATCH  OUT!  Common  mistakes  include  rephrasing  or  repeating  the  observation  or 
conclusion  or  any  other  paragraph.  Also  ensure  that  the  recommendation  follows 
directly  from  the  conclusion. 

USING  THE  LESSON  TEMPLATE 

You  may  have  noticed  that  the  template  described  above,  when  completely  filled  in, 
contains  all  information  required  for  a  LI.  and  therefore  question  the  use  of  this 
template  for  gathering  observations. 

When  capturing  observations,  all  the  five  fields  of  the  ODCR  template  need  not  be 
used.  Only  the  observation  field  is  mandatory.  However,  observation  originators 
should  be  actively  encouraged  to  enter  additional  information  and  supporting  evidence 
in  the  discussion  field.  This  may  increase  the 
efficiency  of  an  LL  process  as  the  LLSO  may  not 
have  to  go  back  to  the  originator  for  more  detail  to 
generate  the  LI. 

As  an  observation  moves  through  an  LL  process, 
additional  information  is  added  to  the  ODCR 
template  fields  so  that,  when  complete,  a  LI  is  the 
result.  Some  examples  of  Lis  in  the  ODCR 
template  format  are  provided  and  discussed  in 
Annex  C  of  this  handbook. 


Tip 

If  writing  down  a  LI,  i.e.  you  already 
have  a  RA,  try  ‘reverse  engineering’ 
the  use  of  the  ODCR  template  by 
writing  down  first  the  observation; 
second  the  recommendation;  third  the 
conclusion  to  support  the 
recommendation;  and  fourth  the 
discussion  needed  to  go  logically  from 
the  observation  to  the  conclusion  and 
recommendation. 
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Annex  C 
Some  LI  Examples 

This  chapter  gives  some  examples  of  Lis  in  the  recommended  ODCR  template.  Text 
boxes  are  used  to  discuss  the  way  that  they  have  been  written. 

JOINT  EXAMPLE 

Title 

Lack  of  training  for  staff  in  Info  Ops. 

Observation 

Info  Ops  cells  at  the  operational  and  tactical  levels 
lacked  adequately  trained  staff. 

Discussion 

Info  Ops  within  NATO  is  a  military  function  to  provide  advice  and  coordination  of 
military  Information  activities.  The  importance  of  appropriately  trained  Info  Ops 
personnel  has  been  raised  to  SHAPE  by  JFC  Naples.  This  issue  was  raised  by  JFC 
Naples  because  current  Military  Committee  policy  is  that  NATO  Info  Ops  training  is 
needed  prior  to  taking  post  and  that  on-the-job  training  is  considered  to  be  an 
unsatisfactory  solution. 

When  Info  Ops  Course  records  were  examined,  it 
showed  that  most  attendees  only  undertook  NATO 
Info  Ops  training  after  arriving  in  post.  The  training 
sometimes  took  place  many  months  after  arrival  in 
post.  Further  investigation  into  why  this  situation 
occurred  revealed  that  there  was  no  stated 
requirement  for  Info  Ops  staff  to  have  completed 
NATO  Info  Ops  training  in  the  relevant  job 
descriptions. 

Conclusion 

One  reason  for  a  lack  of  adequately  trained  Info  Ops 
staff  is  that  training  requirements  for  JFC  Naples  Info 
Ops  staff  are  inadequately  specified  in  relevant  job 
descriptions. 

Recommendation 

Job  descriptions  of  Info  Ops  staff  on  the  JFC  Naples 
peace  establishment  should  be  reviewed  to  include 
in  the  essential  requirements  for  the  post  completion 
of  the  NATO  Info  Ops  training  course.  Action  Body: 

JFC  Naples  Human  Resources. 

MARITIME  EXAMPLE 

Title 

Minehunting  planning  and  evaluation,  fraction  of 
mines  buried. 


Discussion 

The  discussion  starts  with  a  bit  of 
background  and  puts  the 
observation  into  context — who 
raised  the  issue  — and  then 
mentions  the  appropriate  covering 
policy.  It  then  describes  the 
analysis  that  lead  to  the 
identification  of  a  possible  cause  of 
the  issue. 


Conclusion 

Does  not  simply  repeat  the 
observation.  It  describes  the 
overall  finding,  the  cause  of  the 
issue. 


Recommendation 

From  the  observation,  the  obvious 
recommendation  is  that  Alliance 
nations  should  send  adequately 
trained  staff.  However,  in  this 
instance,  the  job  descriptions  did 
not  specify  the  training 
requirements  properly. 


Title 

Title  is  short  and  concise. 


Title 

Title  is  kept  short  and  concise. 

Observation 

Note  that  the  observation 
describes  exactly  what  happened. 
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Observation 

During  minehunting  clearance  operation  trials  with 
different  ships  from  different  nations,  the  reported 
percentage  clearances  varied  significantly. 

Discussion 

Several  minehunters  from  the  NATO  Standing  Group  took  part  in  a  trial  to  investigate 
how  well  the  parameter  percentage  clearance  could  be  evaluated.  Accurate 
percentage  clearance  values  are  essential  to  be  able  to  determine  the  risk  remaining 
from  naval  mines  to  follow-on  shipping. 

The  trials  were  held  over  a  period  of  days  in  the  same  area.  A  number  of  exercise 
mines  were  laid  to  provide  targets  for  the  minehunters.  All  the  participating  units  used 
NATO  doctrine  and  procedures  to  plan  and  evaluate  their  minehunting  operations, 
supported  by  the  standard  NATO  mine  countermeasures  (MCM)  tactical  decision  aid 
MCM  EXPERT. 

The  participating  units  used  MCM  EXPERT  to  plan  the  ordered  clearance  operation 
requiring  a  percentage  clearance  of  96%  of  the  maximum  achievable.  The  units  were 
told  to  carry  out  their  own  assessment  of  the  minehunting  environment  in  the  trials’ 
area,  On  completion  of  the  minehunting  operation,  the  units  used  MCM  EXPERT  to 
evaluate  the  percentage  clearance  achieved  and  report  the  value. 

One  of  the  factors  contributing  significantly  to  the 
widely  varying  reported  percentage  clearance  (from 
48.0%  to  96%)  was  the  different  estimates  of  the 
parameter  “fraction  of  undetectable  mines  due  to 
mine  burial".  For  example,  one  unit  estimated  this 
parameter  as  50%  while  another  unit  estimated  it 
was  0%.  De-briefs  of  the  operations  officers  from  the  units  after  the  trials 
demonstrated  that  this  parameter  was  frequently  being  misinterpreted  as  the  fraction  of 
mine  case  that  was  buried,  rather  than  the  fraction  of  mines  that  were  totally  buried. 

The  relevant  paragraphs  of  the  supporting  NATO  doctrine  were  open  to  different 
interpretations  as  the  wording  was  not  sufficiently  clear. 

Conclusion 

For  the  particular  minehunting  trials  that  were 
examined,  the  wide  differences  in  the  evaluated 
percentage  clearance  achieved  was  mainly  caused 
by  different  interpretations  of  the  parameter  “fraction  of  undetectable  mines  due  to  mine 
burial”;  because  of  the  different  ways  that  it  is  possible  to  interpret  the  explanation  of 
this  parameter  in  the  supporting  doctrine. 

Recommendation 

Re-write  paragraph  xx  through  to  xx  of  ATP-XX  to 
ensure  that  it  is  clear  that  the  parameter  “fraction  of 

undetectable  mines  due  to  mine  burial”  refers  to  the  proportion  of  mines  totally  buried 
and  not  to  the  proportion  of  the  mine  case  that  is  buried.  Action  Body:  Naval 
Minewarfare  Working  Group. 


Recommendation 

Indicates  what  should  be  done  to 
solve  the  problem. 


Conclusion 

The  conclusion  describes  the 
overall  finding,  the  cause  of  the 
issue. 


Discussion 

The  discussion  provides  a  bit  of  a 
story  and  indicates  how  the 
conclusion  was  obtained.  It  is 
logical  to  follow. 


Observation 

Note  that  the  observation  again 
describes  exactly  what  happened. 
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Further  information 

This  was  an  actual  lesson  learned  by  the  NATO  MCM  community.  The  relevant  paragraphs  of  the 
supporting  doctrine  were  re-written  and  another  trial  carried  out  in  order  to  validate  the  re-wording. 

This  second  trial  showed  that  even  the  new  wording  was  open  to  misinterpretation.  As  a  consequence, 
another  rewrite  of  the  offending  paragraphs  was  done  and  a  third  trial  used  to  confirm  that  this  third 
draft  had  indeed  solved  the  issue.  This  third  draft  appears  in  the  Allied  Tactical  Publication  today.  The 
inclusion  of  this  lesson  in  this  Handbook  is  to  demonstrate  with  a  real  example  how  the  process  of 
Validation  works  in  an  LL  process. 


LAND  EXAMPLE 
Title 

Personal  protective  equipment  (PPE)  effectiveness. 

Observation 

Soldiers  lost  belief  in  their  personal  protective 
equipment  (PPE)  and  some  stopped  wearing  it. 

Discussion 

When  interviewed,  many  soldiers  told  anecdotes  about  having  conducted  their  own 
informal  testing  of  their  PPE  by  firing  at  such  items  as  chest  plates  and  helmets.  The 
resulting  damage  to  the  chest  plates  seemed  to  show 
that  the  equipment  would  not  be  effective.  The 
rumours  of  PPE  ineffectiveness  quickly  spread  and 
some  soldiers  were  not  wearing  their  PPE  as  a 
consequence. 

However,  it  was  found  that  some  soldiers  did  not 
understand  that  chest  plates  must  operate  as  a  system 
with  the  frag  vest  in  order  to  function  as  designed:  a 
chest  plate  by  itself  is  not  designed  to  stop  any  specific 
threat.  In  general,  there  appeared  to  be  a  significant  shortage  of  information  available 
to  soldiers  about  the  protective  levels  of  the  equipment  and  how  it  is  designed  to 
operate  together  as  a  system. 

The  responsible  authority  for  dress  standards  and  deportment  is  the  regimental 
sergeant  major  (RSM).  Therefore,  task  force  and  unit  RSMs  are  perceived  as  a  highly 
appropriate  chain  for  delivery,  and  subsequent  enforcement,  of  PPE  protection  policy. 

Conclusion 

Shortage  of  information  about  the  way  that  PPE  works 
as  a  system  led  to  soldiers  conducting  their  own 
misguided  experiments  on  chest  plates  and  drawing 
incorrect  conclusions  about  the  effectiveness  of  the  kit. 

Recommendation 

Develop  briefings  to  better  educate  soldiers  about  their 
PPE  protection  and  how  it  should  be  used.  Action 
Body:  PPE  Procurement  Project  Manager. 

Deploy  briefings  to  soldiers  in-theatre  immediately 
through  RSMs.  Action  Body:  Army  Doctrine  & 

Training. 

Incorporate  briefings  into  basic  training.  Action  Body:  Army  Doctrine  &  Training. 


Conclusion 

Does  not  repeat  the  observation.  It 
describes  the  overall  finding,  the 
cause  of  the  issue. 


Recommendation 

There  are  four  separate 
recommendations  for  two  action 
bodies.  After  the  development  of 
the  necessary  training  material, 
different  actions  are  needed  to 
cover  the  immediate  and  long-term 
requirements. 


Discussion 

Explains  the  context,  why  the 
observed  issue  arose,  and  logically 
explains  the  root  cause  as  a 
shortage  of  information  rather  than 
an  equipment  capability  shortfall, 
which  at  first  sight  may  be  a  more 
plausible  explanation  of  the 
observation. 


Title 

Title  is  kept  short  and  concise. 

Observation 

Observation  succinctly  describes 
exactly  what  the  issue  is. 


41 


Ensure  that  RSMs  are  aware  of  their  responsibilities  to  reinforce  PPE  protection  policy. 
Action  Body:  Army  Doctrine  &  Training. 
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Annex  D 
Interview  Process 


Shamelessly  taken  from  Nick  Milton’s  The  Lessons  Learned  Handbook  (Reference  B), 
pp  41-43. 

A  common  task  for  the  LLSO  is  likely  to  be  gathering  further  information  about 
observations  in  order  to  be  able  to  develop  them  into  LI.  If  there  is  time,  this  is  best 
achieved  through  interviews  with  the  person  or  people  who  submitted  the  original 
observation.  Using  the  following  process  will  give  you  the  best  chance  of  leaving  the 
interview  with  all  of  the  information  you  need. 

Process  Overview 

Think  of  the  parts  of  the  interview  as  parts  of  a  tree  that  you  are  trying  to  explore  during 
the  interview.  The  trunk  is  the  basic  purpose  of  the  interview,  based  on  the  original 
observation.  The  branches  are  all  of  the  issues  surrounding  the  observation  that  you 
would  like  to  explore  more.  Each  branch  then  needs  to  be  explored  to  find  its  root 
causes.  Finally,  you  can  pick  the  fruit  at  the  end  of  the  branch — that  is,  get  the 
interviewee’s  expert  opinion  about  how  the  organization  can  learn  from  the  experience. 

Step  1:  Introduce  the  trunk 

You  will  first  need  to  introduce  yourself  to  the  interviewee  and  explain  what  it  is  you  are 
trying  to  achieve  from  the  interview.  Remind  them  of  the  observation  that  the  interview 
will  be  based  on  and  give  them  an  opportunity  to  say  a  little  about  their  background 
with  respect  to  the  observation. 

Step  2:  Identify  the  branches 

Then  ask  a  number  of  questions  to  identify  what  learning  came  out  of  the  observation. 
These  should  be  “what”  questions  in  the  past  tense  such  as: 

•  What  were  some  of  the  key  issues? 

•  What  were  the  success  factors? 

•  What  worked  well/didn’t  work  well? 

•  What  were  the  challenges  and  pitfalls? 

•  What  would  you  approach  differently  next  time? 

Step  3:  Explore  root  causes 

Then  for  each  of  the  branches  from  step  2  that  seem  interesting,  explore  the  root 
causes  using  “how”  and  “why”  questions  or  using  the  “5  Times  Why”  technique 
described  in  Chapter  4.  Use  open  questions  such  as: 

•  Why  do  you  think  you  were  so  successful? 

•  What  did  you  put  in  place  to  ensure  success? 

•  What  was  missing  that  caused  that  to  happen? 

•  What  makes  you  say  that? 

•  Can  you  explain  how  you  achieved  that? 

•  Can  you  tell  me  about  that? 
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Step  4:  Pick  the  fruit 

When  you  think  you  understand  what  the  learning  is,  get  the  interviewee  to  help  you  to 
identify  some  useful  ways  ahead.  Ask  questions  like: 

•  What  would  be  your  advice  for  someone  else  doing  this  in  the  future? 

•  If  you  were  doing  this  again,  what  would  you  do  differently  next  time? 

•  If  you  could  go  back  in  time  and  give  yourself  a  message,  what  would  it  be? 

Step  5:  Review  your  notes 

When  you  finish  the  interview,  ask  the  interviewee  if  they  mind  checking  your  notes  in  a 
day  or  two.  Put  aside  some  time  immediately  after  the  interview  to  rewrite  your  notes 
in  a  summary  form  that  picks  out  the  most  important  information  the  interviewee  gave 
you.  If  you  use  the  ODCR  format  for  this  and  then  you  will  have  a  draft  LI  ready  for 
review  by  decision  makers  in  your  organization.  Send  your  tidied  notes  to  the 
interviewee  so  that  they  can  check  you  have  captured  their  LI  correctly. 
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Annex  E 
LL  Capability  Checklist 

The  following  items,  presented  as  a  checklist,  are  important  for  an  LL  process  to  deliver 
sustainable  improvement  to  the  efficiency  and  effectiveness  of  an  organization.  This 
checklist  can  be  used  to  assess  the  current  status  of  an  LL  capability  or  to  plan  for 
building  an  LL  capability. 

ORGANIZATION 

□  LL  SOP  in  place,  including  roles  and  responsibilities,  reporting  requirements  and 
staffing  process. 

□  LLSOs  are  able  to  access  key  leaders  required  to  prioritise  lessons,  endorse  RA 
and  task  ABs. 

□  LLSOs  are  trained  in  information  sharing  tools  and  techniques,  LL  and  change 
management  processes,  and  security  classification  procedures. 

□  LLSOs  are  protected  from  double-hatting  and  other  diversions  from  core  tasks. 

□  LLSOs  from  unit  branches  are  internally  trained  and  aware  of  their 
responsibilities  and  reporting  requirements. 

PROCESS 

□  A  process  exists  to  facilitate  the  gathering  of  observations. 

□  The  internal  LL  process  encourages  self-appraisal  and  frank  exchange  of  ideas. 

□  A  process  exists  for  sourcing,  reviewing  and  learning  lessons  derived  from 
external  sources. 

□  A  process  exists  for  submitting  requests  for  assistance,  including  analysis 
support,  to  higher  headquarters. 

□  Staffing  LI  to  Lessons  Learned  is  carried  out  according  to  project  management 
best  practice,  ensuring  leadership  engagement  at  appropriate  stages,  particularly 
at  the  endorsement,  tasking  and  monitoring  stages. 

□  Lessons  are  routinely  and  actively  included  in  planning  operations  and  exercises 
and  in  induction  training. 

□  Lesson  observers  and  originators  are  provided  with  feedback  and  updates  on 
their  contributions. 

TECHNOLOGY 

□  The  LL  process,  and  LL  sharing,  is  supported  by  a  staffing  tool  and  an  archive 
tool  with  search  functionality. 

□  Tools  enable  and  encourage  management  of  LL  information  in  accordance  with 
the  organization’s  information  management  and  security  policies. 
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